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ABSTRACT 


Recent  technological  advancements  now  permit  serious 
consideration  of  embedded  training  as  a  training 
alternative.  To  embed  a  training  subsystem,  decisions  on 
the  composition  of  the  training  package  must  be  made  early 
in  the  developmental  cycle.  Current  methods  of  predicting 
training  effectiveness  do  not  permit  an  effectiveness 
based  comparison  of  training  alternatives  early  in  a  new 
system’s  development.  This  project  presents  a  training 
effectiveness  comparison  method  to  overcome  this  problem. 

The  training  effectiveness  comparison  method  is  a 
seven  step  process  which  identifies  the  tasks  for 
analysis,  screens  tasks  for  embedded  training  candidates 
using  the  U.S.  Army  Research  Institute’s  screening 
process,  structures  the  problem  using  Saaty’s  Analytical 
Hierarchy  Process,  obtains  individual  subject  matter 
expert  training  alternative  rankings  using  the  Technique 
of  Order  Preference  by  Similarity  to  the  Ideal  Solution, 
and  aggregates  those  rankings  using  the  Borda  technique. 
The  project  develops  the  method,  demonstrates  the  method 
using  a  sample  system  (the  next  generation  tank) ,  and 
provides  conclusions  and  recommendations.  The  method  was 
determined  to  be  appropriate  for  preliminary  training 
effectiveness  analyses  of  the  type  conducted  by  the  U.S. 
Army  Training  and  Doctrine  Command. 
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CHAPTER  I.  INTRODUCTION:  THE  PROBLEM. 

Technological  progress  in  training  now  offers 

exciting  opportunities  to  improve  the  U.S.  Army's  ability 

to  effectively  employ  modern,  lethal  weapons  systems  in 

combat.  Advances  in  computational  capacity  and  storage. 

interactive  dynamic  high  fidelity  imagery,  networking, 

software,  courseware,  full  content  voice  recognition  and 

1 

generation,  and  artificial  intelligence  will 
dramatically  improve  the  training  subsystems  in  use  today 
(e.g.  computer  based  instruction,  flight  simulators). 
Moreover,  our  ability  to  apply  this  new  training 
technology  now  is  no  longer  limited  to  conventional 
training  alternatives.  The  advanced  technology  presently 
being  designed  into  new  combat  systems  -  solid  state  fire 
control  computers,  very  high  speed  integrated  circuitry, 
and  electronic  display  screens  -  permits  serious 
discussion  of  a  new  training  subsystem  alternative: 
embedded  training. 

Embedded  training  poses  new  challenges  to  those  who 
develop  combat  systems  and  their  training  subsystems.  But 
before  I  elaborate  on  these  challenges  let  me  define 
embedded  training  and  discuss  its  general  advantages  and 
disadvantages .  The  U.S.  Army  defines  embedded  training 
as:  ‘training  that  is  provided  by  capabilities  designed  to 
be  built  into  or  added  onto  operational  systems  to  enhance 
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and  maintain  the  skill  proficiency  necessary  to  operate 

2 

and  maintain  that  equipment  end  item.*  Embedded 
training  is  a  package  designed  to  train  the  operator  by 
using  the  actual  equipment  rather  than,  for  example, 
externally  based  classroom  instruction  or  aircraft  flight 
simulators.  Embedded  training  has  many  applications. 

Bank  tellers  can  use  intelligent  software  within  the 
computer  terminal  used  to  process  customer  transactions  to 
learn  how  to  process  those  transactions.  Driver’s 
education  of  the  21st  century  may  well  use  an  embedded 
simulation  in  an  actual  automobile  to  provide  a  full  range 
of  driving  situations  a  student  must  master  or  a  licensed 
driver  must  sustain. 

Training  subsystems  which  are  embedded  may  take  one 
of  several  forms.  They  may  be  appended,  adjunct,  or  fully 
integrated  with  the  operational  equipment.  Appended 
subsystems  can  be  quickly  attached.  This  permits  use  of 
the  operational  equipment  to  conduct  the  training 
exercise,  but  the  training  exercise  software  or  other 
hardware  required  to  conduct  the  exercise  is  not  attached 
permanently  or  included  within  the  operational  equipment. 
In  the  driver's  education  example,  the  appended  components 
of  the  training  subsystem  might  be  the  computer  and 
software  package  which  provide  the  situations  and  the 
front  windshield  attachment  which  presents  the  visual 
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simulation  to  the  driver.  The  mounting  hardware  for  the 
windshield  attachment  and  the  electrical  and  data 
connections  to  the  automobile  would  be  installed 
permanently  and  interfaced  with  the  controls  of  the 
automobile.  Adjunct  training  subsystems  are  mounted 
permanently  to  the  operational  system  but  are  separate 
from  the  operational  components.  The  computer  which 
provides  the  situation  package  in  the  automobile  example 
might  be  mounted  permanently  somewhere  within  the 
automobile  and  already  attached  to  the  electro-mechanical 
interfaces  required  to  process  information  between  the 
training  simulation  (situations)  and  the  driver.  However 
the  computer  would  remain  separate  -  its  sole  function 
would  be  to  act  as  part  of  the  training  subsystem.  A 
training  capability  may  also  be  fully  integrated  within 
the  operational  equipment.  In  the  case  of  our  future 
automobile,  the  onboard  computer  (which  might  control 
interior  temperature,  fuel  efficiency  and  fuel  dispensing 
brake  operation,  and  position  navigation)  would  also 
contain  the  training  software  necessary  to  operate  the 
situation  simulation.  This  fully  integrated  embedded 
training  subsytem  would  use  many  of  the  same  electronic 
and  data  circuits  required  for  normal  automobile 
operation . 

There  is  another  point  to  be  made  about  embedded 
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training.  Embedded  training  is  more  than  the  simple 

presentation  of  information.  It  must  provide  performance 

assessment  to  de+ermine  the  proficiency  level  of  the  user. 

feedback  to  the  user  to  improve  performance  or  reinforce 

correct  performance,  and  record  keeping  to  manage  the 

3 

training  proficiency  progression  of  the  user.  This 
three  part  requirement  may  be  provided  by  the  training 
subsystem  itself  or  may  be  provided  by  an  'off  line" 
instructor  or  facilitator  designated  to  observe  the 
training . 

The  use  of  embedded  training  can  Improve  dramatically 
the  training  readiness  of  soldiers,  crews,  and  units. 
Embedded  training  has  several  distinct  advantages.  First, 
training  embedded  in  the  combat  system  provides  a  training 
subsystem  which  is  available  concurrently  with  the 
fielding  of  the  new  system.  Training  devices  or 
simulators  for  reasons  of  cost  or  available  training 
technology  frequently  have  been  fielded  after  the  new 
combat  system  has  arrived  at  the  using  unit.  Because 
embedded  training  is  designed  into  the  combat  system  at 
the  earliest  engineering  stages,  this  training  subsystem 
can  be  tested  and  produced  at  the  same  time  as  the  combat 
system.  Second,  embedded  training  will  permit  instruction 
on  how  to  operate  the  tank  without  the  large  instructor 
cadre  required  today.  This  training  will  provide  a 
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training  program  for  soldiers  who  must  use  different  tanka 

from  either  prepoaitioned  atock8  (POMCUS)  or  war  reserve 

during  periods  of  conflict  when  large,  fully  trained 

instructor  groups  are  not  available.  Third,  embedded 

training  provides  a  readily  available  training  package  for 

austainment  training  in  units.  Fourth,  embedded  training 

can  provide  a  training  management  (record  keeping) 

capability  that  will  relieve  this  type  of  administrative 

burden  from  unit  trainers.  This  capability  may  ultimately 

provide  more  objectivity  in  the  readiness  reporting 

ay8tam.  Fifth,  embedded  training  can  help  standardize 

training  across  the  force,  regardless  of  the  geographic 

location  or  major  Army  command  (MACOM)  of  the  soldier.  It 

does  this  because  the  lesson  content  and  performance 

standards  originate  from  one  source  and  are  part  of  the 
4 

combat  system.  Moreover,  it  may  be  easier  to 
incorporate  performance  assessment  into  an  integrated 
embedded  training  system. 

Embedded  training  is  not  without  its  disadvantages. 
First,  the  costs  can  be  high.  The  costs  include  both 
component  acquisition  and  lesson  development  costs.  These 
lesson  development  costs  continue  as  doctrine  evolves  over 
the  life  of  the  combat  system.  These  costs  are  often 
included  in  the  total  procurement  cost  of  the  combat 
system.  This  means  a  developer  with  a  fixed  budget 
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may  have  to  accept  a  ?  ess  effective  training  subsystem 

because  the  item  cost  now  makes  achieving  the  desired 

quantity  impossible.  Second,  embedded  training  requires 

more  frequent  use  of  the  combat  system.  This  use  makes  it 

more  difficult  to  achieve  the  required  level  of 

reliability,  availability,  and  maintainability  of  the 

system.  Additional  ruggedness  must  be  designed  into  the 

parts  of  the  system  used  for  training  to  ensure  the  system 

can  maintain  its  required  operational  rate.  For  those 

parts  of  the  training  subsystem  that  are  embedded  fully 

with  the  operational  hardware,  designers  must  ensure  that 

failure  of  the  training  subsystem  does  not  affect 

5 

performance  of  the  operational  subsystem.  Finally, 

since  embedded  training  requires  advanced  technology, 

there  are  technical  risks  associated  with  choosing  a 

technology  that  will  be  available  in  time  for  production. 

Failure  to  advance  a  key  technology  may  change  the 

embedded  training  approach  dramatically  .  Of  course,  this 

is  a  risk  to  the  operational  system  as  well,  but  in  the 

past,  training  developers  frequently  relied  on  proven 

technology  to  develop  training  devices  for  existing 
6 

systems . 

The  U.S.  Army  has  recognized  the  potential  training 
opportunities  embedded  training  has  to  offer.  For  this 
reason,  Army  policy  requires  that: 
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*An  embedded  training  capability  be 
thoroughly  evaluated  and  considered  as  the 
preferred  alternative  among  other  approaches 
to  the  incorporation  of  training  subsystems 
in  the  development  and  follow  on  Product 
Improvement  Programs  of  all  Army  material 
systems . ’ 7 

This  means  that  embedded  training  is  now  the  training 
alternative  of  choice.  Moreover,  the  reasons  that 
embedded  training  is  not  selected  (the  Army  does  recognize 
that  embedded  training  is  not  appropriate  for  all  tasks) 
for  a  particular  portion  of  the  training  package  will  be 
discussed  during  each  major  milestone  decision  review  of 
the  new  operational  system. 

The  formal  process  for  the  study  of  training 
alternatives  is  the  U.S.  Army  Training  and  Doctrine 
Command  Training  Effectiveness  Analysis  (TEA)  system. 
Developmental  training  effectiveness  analyses  (DTEA) ,  one 
of  two  major  subcategories  of  the  TEA  system,  are  the 
analyses  which  look  at  the  application  of  embedded 
training  to  the  training  subsystem.  The  DTEA  is  comprised 
of  the  preliminary  training  effectiveness  analysis  (PTEA) . 
the  cost  and  training  effectiveness  analysis  (CTEA) ,  and 
the  training  development  study  (TDS) .  The  PTEA 
contributes  to  the  development  of  training  strategies  for 
a  new  hardware  system.  It  generally  assesses:  who  is  to 
be  trained;  what  skills  are  to  be  trained;  and  how,  when, 
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and  where  training  will  be  accomplished.  The  CTEA, 

conducted  after  the  PTEA,  investigates  the  comparative 

effectiveness  and  cost  of  training  alternatives  developed 

for  a  new  hardware  system.  It  includes  an  analysis  of  the 

levels  of  proficiency  attained  by  the  different  training 

al ternatives ,  an  analysis  of  the  cost  associated  with 

these  alternatives,  and  a  cost  effectiveness  tradeoff 
8 

analysis  . 

The  effectiveness  study  of  embedded  training  is  a  new 
challenge  to  training  analysts  because  decisions  about  the 
incorporation  of  this  alternative  must  occur  early  in  the 
development  cycle  of  a  new  system.  Other  training 
alternatives  can  be  designed  effectively  only  after  much 
is  known  about  the  final  design  of  the  new  system. 

Embedded  training,  in  order  actually  to  be  embedded,  must 
be  incorporated  into  the  system  design  early  to  avoid  a 
significant  redesign  effort  later  in  the  developmental 
cycle.  These  necessary  early  decisions  are  in  fact  an 
advantage  of  embedded  training.  They  force  the 
operational  designer  of  the  system  to  consider  the  effect 
the  design  will  have  on  the  requirement  to  train  the 
operator  how  to  use  it.  This  is  so  because  the  next  step 
will  be  to  consider  what  type  of  training  package  will  be 
required  to  train  the  system  just  designed. 

An  investigation  into  the  effectiveness  of  a  training 
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alternative  is  not  so  difficult  later  in  the  design  cycle 
when  an  operational  system  equipped  with  an  embedded 
training  package  is  available  for  testing.  However,  it  is 
very  difficult  to  evaluate  a  proposed  embedded  training 
package  in  the  early  stages  (PTEA)  of  system  development. 
It  is  difficult  because  little  empirical  data  exist  from 
which  to  analyze  and  subsequently  decide  upon  a  training 
package  which  then  must  go  to  the  full  scale  engineering 
stage . 

There  are  several  methods  in  use  today  which  are 
designed  to  predict  the  effectiveness  of  either  training 
programs  or  training  devices.  The  principal  methods  are 
discussed  in  Chapter  II.  These  methods  are  not  suited  for 
the  required  early  investigation  of  embedded  training  for 
two  reasons.  First,  they  either  do  not  include  embedded 
training  as  a  training  medium,  method,  or  alternative. 
Second,  they  often  require  the  very  data  which  we  have 
just  seen  is  not  normally  available  early  in  the 
developmental  cycle. 

The  problem,  therefore  is  that  current  training 
effectiveness  prediction  methods  do  not  permit  a 
comparison  of  embedded  training  to  other  training 
alternatives  early  in  the  developmental  cycle  of  a  system. 
The  purpose  of  this  project  has  been  to  develop  such  a 
method.  This  method  will  be  applied  during  the 
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preliminary  training  effectiveness  analysis  stage.  It  is 
here  where  training  strategies  are  developed  and  where 
candidates  (by  function  or  task)  for  embedded  training  are 
ident i f i ed . 

Research  into  embedded  training  evaluation  began 
several  years  ago.  A  method  has  been  developed  to 
identify  candidate  tasks  for  embedded  training.  The  U.S. 
Army  Research  Institute  generated  a  screening  process  to 
select  embedded  training  candidates  starting  with  a  list 
of  tasks  which  must  be  trained  to  operate  the  new  system. 
What  remains  to  be  established  is  a  method  for  determining 
the  training  effectiveness  of  these  candidates  as  compared 
to  other  training  subsystem  alternatives. 

This  project  is  organized  to  establish  such  a  method 

in  three  parts.  First,  Chapters  I  through  IV  develop  the 

method  by:  identifying  the  problem,  providing  a  summary 

of  principal  methods  available  today,  discussing  the  ARI 

9 

screening  process  ,  and  detailing  the  method's 
development.  Second,  Chapters  V  through  VII  demonstrate 
the  model  using  a  sample  system.  This  is  accomplished  by 
screening  the  gunner’s  position  tasks  of  the  Future 
Armored  Combat  System  -  the  next  generation  tank,  setting 
up  the  problem,  and  determining  final  rankings.  Third, 
Chapters  VIII  and  IX  provide  conclusions  and 
recommendations .  Finally,  the  appendices  provide  input 
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data,  background  information,  and  the  organization  of  the 
pro  j  ect . 

The  object  of  this  effort  is  to  go  beyond  the 
screening  process  in  an  effort  to  provide  the 
decisionmaker  critical  training  effectiveness 
information.  This  additional  information  increases  the 
chance  of  selecting  the  best  training  alternative  (in 
terms  of  effectiveness)  at  the  critical  point  early  in  the 
development  cycle.  This  training  alternative  ranking 
method  can  be  used  by  all  the  proponent  schools  of  the 
U.S.  Army  Training  and  Doctrine  Command  as  they  perform 
preliminary  training  effectiveness  analyses  on  their 
developing  combat  systems. 
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CHAPTER  II.  CURRENT  METHODS  OF  PREDICTING  TRAINING 
EFFECTIVENESS. 

There  are  several  methods  which  have  been  developed 
to  estimate  the  effectiveness  of  a  training  program.  The 
purpose  for  this  section  is  twofold.  First,  the  criteria 
for  the  training  alternative  comparison  method  developed 
in  Section  IV  are  obtained  from  the  substantial  research 
conducted  to  establish  the  present  methods.  Second,  these 
models  are  shown  to  demonstrate  that  they  do  not  resolve 
the  problem  presented  in  Chapter  I.  Therefore,  a  new 
method  is  required.  The  four  key  models  are  the  Training 
Efficiency  Estimation  Model  (TEEM) ,  Analogous  Task  Method, 
TRAINVICE  IV,  and  Training  Consonance  Analysis.  This 
section  provides  a  summary  of  how  these  models  work. 

The  Training  Efficiency  Estimation  Model  was 

10 

developed  by  Jorgensen  and  Hoffer  in  1978.  This  model 
generates  an  estimated  training  program  for  a  developing 
system  through  task  analysis,  selection  of  training  media 
and  methods,  and  identification  of  information  content  and 
structure.  For  each  alternative  training  program 
established  an  efficiency  ratio  is  calculated.  This  ratio 
represents  the  training  efficiency  value  of  the  program 
when  subjected  to  real  world  constraints  divided  by  the 
training  efficiency  value  of  the  same  program  with  no 
constraints.  This  efficiency  ratio  can  then  be  compared 
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to  alternative  training  programs  to  predict  the  most 
effective  program  for  a  developing  system. 

The  TEEM  analysis  begins  with  a  comparison  of  a  task 
required  by  the  developing  system  (e.g.  engage  targets 
with  the  main  gun  from  the  gunner’s  station)  to  three 
general  categories  of  media  related  variables  -  stimulus, 
response,  and  feedback.  Examples  of  these  variables  are: 
stimuli  -  medium  of  stimuli  presentation,  visual  form, 
visual  movement:  response  -  response  mode  of 
implementation,  intensity  of  response,  required  response 
rate;  and  feedback  -  medium  of  feedback  presentation, 
source  of  feedback,  type  of  feedback.  A  fourth  set  of 
variables,  functional  context  variables,  are  also  compared 
to  the  task.  These  four  categories  of  variables  (39 
stimulus,  16  response,  17  feedback,  and  13  functional 
context)  are  evaluated  to  determine  if  they  apply  to  the 
task.  A  1  is  awarded  to  variables  that  apply  to  the 

task.  A  0  is  awarded  to  variables  that  do  not  apply. 

The  next  step  in  the  TEEM  analysis  is  to  determine 

what  training  medium  best  fits  the  particular  task  under 

consideration.  First,  each  group  of  the  three  media 
related  variables  (stimulus,  response,  and  feedback)  are 
compared  against  38  items  of  training  equipment  and 
materials  (  e.g.  case  study  folders,  flash  cards, 
programmed  texts,  computer,  etc.) .  Again,  1  is  awarded 
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when  the  training  equipment  or  material  matches  the 
stimulus,  response,  or  feedback  variable.  Second,  using 
only  those  media  variables  initially  identified  (first 
step)  as  relevant  to  the  task  under  consideration,  a  total 
score  for  each  of  the  different  types  of  training 
materials  is  computed.  The  training  material  with  the 
highest  score  is  selected  (in  each  of  the  three  media 
related  variable  categories).  Ties  are  broken  by 
selecting  the  least  expensive  training  material. 

The  third  step  is  to  select  the  method  of  training. 
This  is  accomplished  by  comparing  23  methods  of  training 
(e.g.  lecture,  conference,  demonstration,  role  playing) 
with  the  13  functional  context  variables  (e.g.  role  of 
element,  function  performed  in  the  role,  physical  context, 
psychological  impact) .  Again  1  is  awarded  when  the  method 
matches  a  functional  context  variable  for  the  task  under 
consideration.  Then,  using  only  those  functional  context 
variables  initially  identified  (first  step)  as  appropriate 
to  the  task  under  consideration,  a  total  score  for  each  of 
the  different  training  methods  is  selected.  The  method 
with  the  highest  score  is  selected. 

The  final  step  in  the  TEEM  analysis  is  to  determine 
the  optimal  mix  of  media  for  all  the  tasks  that  support 
the  developing  system  or  to  compare  alternative  mixes  of 
media  (selected  for  reasons  of  cost  or  availability). 
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This  comparison  measure  is  an  efficiency  ratio  -  the 

result  of  dividing  a  program  composed  of  affordable  media 

by  a  program  which  provides  the  best  media  for  each  of  the 

tasks  in  the  system.  This  ratio  is  obtained  by  first, 

forming  a  matrix  for  each  of  the  three  media  related 

variables  which  contain  the  media  selection  scores  (step 

two)  for  each  task  considered.  Next,  add  the  largest 

values  for  each  task  in  the  matrix.  This  produces  a  total 

score  for  the  stimulus,  response,  and  feedback  matrices 

which  are  then  summed  to  obtain  the  overall  measure.  This 

score  represents  the  highest  possible  media  effectiveness 

of  the  tasks  which  support  the  developing  system  and  is 

the  denominator  used  for  all  efficiency  ratio  (alternate 

program)  calculations.  The  numerator  for  the  ratio  is 

obtained  by  summing  the  scores  of  the  media  used  in  the 

alternative  training  programs.  These  ratios  are  then 

compared  to  determine  which  alternative  training  program 

(alternative  media  combination)  has  the  highest  efficiency 

1 1 

ratio  (closest  to  one).  The  most  training  effective 
program  is  judged  on  its  cost  implications  to  develop  a 
final  alternative  training  program  recommendation. 

The  second  method  to  estimate  the  effectiveness  of  a 
training  program  is  the  Analogous  Task  Method.  This 
method  compares  tasks  of  the  developing  system  to  tasks 
from  a  fielded  system.  This  comparison  methodology 
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permits  the  analyst  to  predict  the  success  of  the  training 

plan,  to  determine  likely  problem  areas  within  the  plan, 

and  to  develop  alternative  approaches  for  the  problem 
12 

areas . 

The  analysis  methodology  first  rates  the  task  under 
consideration  (the  fielded  system  -analogous-  task  or  the 
developing  system  task)  in  terms  of  several  task 
descriptor  variables.  These  are  action  verb  type, 
initiating  cues,  general  cue  characteristics,  visual  cue 
characteristics,  auditory  cue  characteristics,  categories 
of  feedback  information,  and  feedback  characteristics. 

Each  task  will  contain  a  certain  number  of  these  task 
descriptor  variables.  Analogous  tasks  are  selected  for 
use  in  the  next  analysis  step  if  their  task  descriptor 
variables  closely  match  the  developing  system  task.  The 
methods  of  training  the  selected  analogous  tasks  forms  the 
analogous  training  program.  This  training  program 
provides  a  good  estimation  of  the  training  program  for  the 
developing  system. 

The  final  step  in  the  analogous  task  method  is  to 
predict  the  effectiveness  of  the  training  program  for  the 
developing  system.  This  prediction  is  established  by 
obtaining  performance  data  from  the  analogous  training 
program  developed  in  the  first  step.  Soldier 
qualification  test  results,  end  of  course  performance 
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exams,  and  unit  training  results  are  all  used  to  establish 
the  effectiveness  of  the  program. 

The  third  method  used  to  estimate  the  effectiveness 

of  a  training  program  is  the  TRAINVICE  method.  The 

TRAINVICE  model  has  evolved  between  1976  and  1980  into 
13 

four  versions.  TRAINVICE  I,  developed  by  Wheaton, 
Fingerman,  Rose,  and  Leonard  in  1976  predicted  the 
effectiveness  of  a  device  by  examining,  at  the  subtask 
level,  the  fit  of  a  proposed  device  in  terms  of  task 
commonality,  physical  similarity,  functional  similarity, 
learning  deficit  and  training  technique.  TRAINVICE  II, 
developed  by  the  office  of  the  Project  Manager  for 
Training  Devices  (an  Army  Material  Command  Agency)  in  1979 
is  a  modification  of  the  original  TRAINVICE  I  model.  Both 
models  share  the  physical  similarity  and  functional 
similarity  analyses  (although  methods  of  determining  a 
rating  are  slightly  different).  TRAINVICE  II  does  not, 
however,  assess  the  training  techniques  used  in  the  device 
and  TRAINVICE  II  penalizes  a  device  for  including  features 
which  are  not  required.  TRAINVICE  III,  developed  by  Narva 
in  1979,  revised  the  TRAINVICE  I  approach  in  an  attempt  to 
make  the  procedure  more  practical  and  flexible.  This 
model  evaluates  a  proposed  device  in  terms  of  coverage 
requirement,  coverage,  training  criticality,  training 
difficulty,  physical  characteristics,  and  functional 


characteristics  . 


The  final  variant,  TRAINVICE  IV,  is  a 


modification  of  TRAINVICE  III  and  was  developed  by  Swezey 
and  Evans  in  1980.  In  this  version  training  criticality 
was  changed  to  training  proficiency  and  some  of  the  rating 
scales  were  modified  to  provide  more  guidance  to  users. 

In  each  model  an  e f f ec t i veness  formula  was  used  to  compare 
a  proposed  device  to  training  on  the  actual  piece  of  the 
equipment . 

The  1985  application  of  these  models  to  Armor 

training  devices  by  Harris,  and  others,  suggests  that  'the 

simplest  prediction  method  (TRAINVICE  II)  is  the  one  to 

use  since  its  predictions  agree  with  the  more  complicated 
14 

methods.'  Therefore,  the  TRAINVICE  methodology  is 
presented  in  detail  using  the  TRAINVICE  II  approach. 

TRAINVICE  II  uses  ratings  on  device  characteristics 
and  personnel  and  training  requirements  to  establish  an 
effectiveness  index  for  the  device  or  device  concept. 

This  index  reflects  the  degree  to  which  the  device 
corresponds  to  the  operational  equipment,  the  extent  of 
the  training  required,  and  the  trainee  ability  level. 
Device  characteristics  are  evaluated  in  terms  of  task 
commonality,  physical  similarity  and  functional 
similarity.  The  task  commonality  analysis  evaluates 
whether  or  not  a  device  trains  a  subtask.  A  task 
numerical  score,  for  each  device,  is  determined  first  by 
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assigning  a  1  to  subtasks  that  are  covered  by  the  device 
(or  a  0  to  subtasks  that  are  not  covered)  and  second  by 
summing  the  subtask  scores  and  dividing  this  subtotal  by 
the  sum  of  the  total  number  of  subtasks  plus  the  number  of 
elements  the  device  has  which  are  not  required  to  train  a 
required  subtask.  The  formula  is  expressed  as: 

Task  Commonality  = _ f_of_l_^s_for_devi.ce _ 

Index  #  of  req  subtasks  +  *  of  extra  elements 

The  next  step  is  to  obtain  a  rating  of  the  physical 
similarity  between  the  device  and  the  operational 
equipment.  This  is  accomplished  by  rating,  for  each  task, 
the  degree  of  fidelity  between  the  operational  equipment’s 
displays  and  controls  and  the  device.  The  rating  is  based 
on  a  four  point  scale:  3  -  identical;  2  -  similar;  1  - 
dissimilar;  0  -  missing.  Task  ratings  for  each  device  are 
then  determined  by  adding  the  similarity  scores  together 
and  then  dividing  this  total  by  three  times  the  number  of 
controls  and  displays  plus  the  number  of  controls  or 
displays  found  on  the  device  but  are  not  required  by  the 
actual  equipment.  The  formula  is  expressed  as: 

Physical  Similarity  = _ i mi  1  ar  1  ty _rat i^ngs _ 

Index  3(#  of  req  controls  &  displays)  + 

(*  of  extra  controls  &  displays) 

The  third  step  is  to  assess  the  functional  similarity 

of  the  device  to  the  operational  equipment.  This 
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assessment  evaluates  whether  or  not  the  information  flow 
between  the  operator  and  the  device  and  the  operator  and 
the  operational  equipment  is  the  same.  As  in  the  physical 
similarity  analysis,  the  functional  similarity  analysis 
rates  the  controls  and  displays  that  are  required  to 
perform  a  certain  task  on  a  four  point  scale:  3  - 
identical;  2-  similar;  1  -  dissimilar;  0  -  missing.  The 
overall  rating  is  established  in  the  same  manner  as  in  the 
physical  similarity  index  and  uses  the  same  formula. 

The  fourth  step  determines  the  student’s  baseline 
knowledge  and  skill  before  training  on  the  task  required 
for  operation  of  the  equipment  and  compares  this  knowledge 
and  skill  to  what  is  required  for  successful  performance 
of  the  task.  Each  task  is  broken  down  into  its  component 
skills  and  knowledge.  Each  Skill  and  knowledge  is  then 
evaluated  on  a  rating  scale  of  0  (no  training)  to  4 
(complete  understanding)  in  each  of  the  two  categories: 
before  training  and  after  training.  A  Skill  and  Knowledge 
Requirements  Index  is  then  obtained  by  summing  the 
numerical  differences  between  after  training  and  before 
training  and  dividing  this  sum  by  four  times  the  number  of 
skills  and  knowledge  in  the  task.  The  formula  is: 

Skill  &  Knowledge  = _ Total^_difference_score _ 

Requirements  Index  4(#  of  skills  &  knowledge) 

The  fifth  step  involves  estimating  the  length  of  time 


(an  estimate  of  difficulty)  it  will  take  to  train  the  task 
on  the  operational  equipment.  Again  a  0  (requires  no 
training  time)  to  4  (requires  as  much  time  to  train  as  any 
task  element)  is  used  to  rate  each  task  element.  A  task 
training  difficulty  index  is  then  obtained  by  summing  the 
task  element  ratings  and  dividing  this  sum  by  four  times 
the  number  of  task  elements.  The  formula  is  expressed  as: 

Task  Training  =  To tal _train ing_di f f icul ty_score_ 

Difficulty  Index  4(#  of  task  elements) 

In  the  sixth  step,  the  analyst  calculates  an 
effectiveness  score  for  each  task.  For  each  device  the 
average  of  the  task  commonality  (TC) ,  physical  similarity 
(PS),  and  functional  similarity  (FS)  scores  is  obtained. 
This  is  obtained  by:  (TC  +  PS  +  FS)/3.  The  personnel  and 
training  requirements  scores  are  independent  of  a 
particular  device.  Therefore  each  device  is  awarded  the 
same  personnel  and  training  requirements  score.  The  task 
score  is  obtained  by:  (SKR  +  TTD)/2,  where  SKR  is  the 
skill  and  knowledge  requirements  index  and  TTD  is  the  task 
training  difficulty  index. 

The  final  step  is  to  obtain  and  compare  the  overall 
device  effectiveness  index  for  each  device.  The  product, 

( (TC+PS+FS) /3) ( (SKR+TTD) / 2) ,  is  obtained  for  each  task. 
These  products  are  then  summed  over  all  the  tasks.  This 
sum  is  then  divided  by  the  sum  (over  all  the  tasks)  of  the 
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personnel  and  training  requirements  score  ( (SKR+TTD) /2) . 
This  quotient  is  then  multiplied  by  an  excess  task 
(feature)  penalty  quotient.  The  final  formula  is  given 
by  : 

Training  Device  Effectiveness  Index  = 

§y?D_°l_llIC  +  PS  +  FS  V3)J_(SKR  +  TTD)_/2)_  X  *_of_reg_ tasks 

Sum  of  (SKR  +  TTD) /2  (#  of  req  tasks) + 

(#  of  extra  features) 

These  final  scores,  which  range  from  0  to  1 ,  are  then 

compared.  The  device  or  device  concept  with  the  highest 

score  has  the  highest  effectiveness  potential. 

The  fourth  method  to  estimate  the  effectiveness  of  a 

training  program  is  Training  Consonance  Analysis.  This 

15 

analysis,  developed  by  Hawley  and  Thomason  (1978)  , 

uses  a  training  consonance  ratio  as  a  predictor  of 
training  effectiveness  for  a  device  or  device  concept. 

The  analysis  begins  by  using  the  initial  analysis 
methodology  in  the  Training  Efficiency  Estimation  Model. 

t 

Each  task  is  compared  with  stimuli,  response,  information 
feedback  logic,  and  functional  context  variables.  A  1  is 
awarded  to  the  variable  if  it  applies  to  the  task,  a  0  if 
it  does  not.  This  analysis  is  called  the  task 
description.  The  training  description  is  obtained  by 
describing  each  task  in  terms  of  both  stimulus,  response, 
and  feedback  media  found  in  a  list  of  training  equipment 
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and  in  terms  of  training  methods  found  in  a  list  of 
training  methodologies.  These  data  are  then  entered  into 
a  computer  program  which  compares  the  task  description 
(variables  which  describe  the  task)  and  the  training 
description  (variables  which  describe  the  media  or  method 
match  to  the  proposed  device)  .  The  output  provides  both 
the  number  of  training  deficiencies,  excesses,  and 
redundancies  and  a  training  consonance  ratio  (TCR) .  This 
ratio,  TCR,  is  the  quotient  of  the  number  of  matches 
between  the  task  characteristics  and  training  description 
divided  by  the  total  number  of  task  characteristics  in 
that  particular  task.  Generally  speaking  a  TCR  of  less 
than  .6  is  not  acceptable.  The  program  will  compute  an 
overall  TCR  for  the  training  program  alternative.  The 
program  with  the  highest  TCR  is  usually  accepted  (although 
further  task  level  analysis  can  occur) . 

The  training  effectiveness  comparison  method 
developed  in  this  project  is  designed  to  overcome  the  gaps 
found  in  the  models  available  today.  First,  they  do  not 
consider  embedded  training  as  a  training  alternative. 

This  is  understandable  since  the  models  were  designed 
before  embedded  training  became  a  practical  training 
alternative.  TEEM  and,  by  extension,  TCR  show  the 
greatest  promise  for  accepting  embedded  training  into 
their  present  model  structure.  However,  this  is  not  a 
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simple  matter.  Embedded  training  can  be  viewed  as  both  a 
training  medium  (e.g.  flash  cards,  audio  disc  system, 
television  system)  and  as  a  training  method  (e.g. 
lecture-team  teach,  programmed  instruction,  panels). 
Additional  research  must  be  conducted  to  establish  the 
best  way  to  incorporate  embedded  training  into  TEEM. 
Further,  TEEM  and  TCR  often  result  in  a  training 
alternative  recommendation  that  consists  of  several 
different  training  medium.  The  training  effectiveness 
comparison  method  permits  a  focused  comparison  of  specific 
medium  (up  to  9  versus  38)  without  permitting  the 
introduction  of  undesired  medium.  Second,  most  present 
day  tasks  are  not  trained  using  embedded  training. 
Therefore,  the  Analogous  Task  Method  cannot  be  used  to 
compare  today’s  training  alternatives  to  embedded  training 
since  it  relies  on  present  training  methods  to  suggest 
training  alternatives  for  analogous  tasks.  Moreover,  ATM 
does  not  account  for  new  tasks  which  are  not  analogous  to 
the  fielded  system.  As  more  systems  are  fielded  with 
embedded  training  subsystems  the  Analogous  Task  Method 
will  become  a  more  important  training  effectiven ess 
comparison  method.  Finally,  the  TRAINVICE  series  of 
models  could  be  used  to  compare  embedded  training  to  other 
training  devices.  Unfortunately,  the  TRAINVICE  series  of 
models  rely  on  subtask  information  that  is  not  available 


24 


at  the  early  stage  of  the  development  cycle.  Therefore,  a 

detailed  TRAINVICE  comparison  cannot  be  performed  until 

after  the  initial  embedded  training  decision  is  required. 

Specific  research  into  the  unique  nature  of  embedded 

training  has  begun.  This  U.S.  Army  Research  Institute 

sponsored  effort  has  generated  procedures,  described  in 

ten  volumes,  which  enable  new  systems  developers  to 

incorporate  embedded  training  into  their  operational 
16 

systems.  Chapter  III  presents  a  summary  of  that 
portion  of  the  research  effort  that  is  appropriate  to  this 
project  -  the  embedded  training  candidate  task  screening 
method . 
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CHAPTER  III.  EMBEDDED  TRAINING  -  CURRENT  SCREENING 
METHODS . 

Roth,  Sullivan,  and  Ditzian  present  a  four  phased 

process  by  which  an  analyst  can  determine  what  tasks  on  a 

new  system  are  likely  candidates  for  embedded  ^raining. 

Phase  I  defines  the  tasks  that  must  be  performed  to 

operate  the  new  piece  of  equipment.  Phase  II  develops 

behavioral  performance  objectives  and  identifies  how  each 

performance  objective  will  be  evaluated  for  the  tasks 

identified  in  Phase  I.  Phase  III  nominates,  on  the  basis 

of  criticality  and  perishability  candidate  tasks  for 

embedded  training.  Additionally  during  this  phase  the 

analyst  assesses  the  ability  to  implement  the  nominated 

task3 .  Finally,  Phase  IV  provides  a  procedure  by  which 

the  analyst  can  document  the  data  obtained  during  Phases 
17 

I , 1 1 ,  and  III. 

Phase  I  is  a  nine  step  process  used  to  identify  the 
tasks  that  personnel  must  be  able  to  accomplish  to  operate 
the  system.  The  first  step  is  to  gather  information  on 
the  new  system  through  available  literature  or  information 
from  subject  matter  experts.  Next,  the  analyst 
identifies  the  job  positions  involved  in  the  operation  of 
the  system.  Third,  the  analyst  identifies  the  missions 
(e.g.  attack,  defend)  which  must  be  performed  by  the  new 
system.  At  the  next  step,  the  analyst  establishes  a 
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computer  database  and  enters  the  data  available  thus  far. 

The  fifth  step  requires  the  analyst  to  identify  the 

subcomponents  of  the  missions  identified  in  the  third 

step.  This  is  accomplished  by  a  search  through  the 

available  doctrinal  literature  and  discussions  with 

subject  matter  experts.  Next, the  analyst  identifies 

mission  subcomponents  which  are  common  to  several 

missions.  This  is  a  bookkeeping  step  used  to  ensure  that 

only  unique  subcomponents  are  analyzed  in  later  steps.  In 

the  seventh  step,  each  task  which  is  performed  while 

executing  a  subcomponent  of  a  mission  is  identified  (e.g. 

engage  targets  with  the  main  gun  from  the  gunner’s 

position).  The  eighth  step  is  similar  to  the  sixth.  In 

this  step  the  analyst  identifies  the  tasks  which  are 

common  to  several  mission  subcomponents.  Finally  the 

analyst  identifies  the  crew  position  responsible  for  each 

task.  The  results  of  steps  five  through  nine  are  entered 

in  the  database  during  each  of  these  steps.  The  analyst 

is  now  ready  for  the  detailed  task  analysis  of  Phase 
18 

II  . 

Phase  II  is  an  optional  phase  which  depends  on  when 
the  analysis  takes  place  in  the  development  of  the  new 
system.  Roth,  and  others,  point  out  that  when  using  this 
methodology  for  identifying  embedded  training  requirements 
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early  in  the  system  life  cycle,  specific  data  required  for 

detailed  task  analysis  likely  will  not  be  available.  In 

this  case  the  analyst  can  use  the  task  level  of 

information  and  proceed  on  to  phase  III.  Ultimately  the 

analyst  should  return  to  Phase  II  when  the  information 
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becomes  available. 

Phase  II  is  a  two  step  procedure.  First,  each  unique 

task  is  analyzed  to  determine  exactly  how  the  operator 

performs  the  task  and  what  stimuli  is  provided  by  the 

equipment  during  task  performance.  This  step  identifies 

the  component  subtasks,  the  required  behavioral  action, 

the  amount  of  memorization  required,  and  whether  or  not  a 

subtask  is  mission  or  training  (performed  only  in  a 

training  environment)  related.  The  second  step  identifies 

the  performance  measure  variables  to  be  used  to  assess  how 

well  the  tasks  identified  in  the  first  step  are 

completed.  Some  examples  of  these  variables  are:  time  or 

speed  of  performance,  accuracy  or  error  rate,  safety 

considerations,  process  measures,  and  product  measures. 

The  specific  numbered  value  of  a  variable  is  not  important 

at  this  stage.  Rather,  the  intent  is  only  to  identify  the 
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assessment  criteria. 
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The  heart  of  the  Roth,  and  others,  procedure  is  the 

third  phase.  This  phase  identifies  those  tasks  which  are 

good  candidates  for  embedded  training.  First,  tasks  are 

nominated  if  they  are  critical  to  mission  success  or  if 

the  soldier  can  easily  lose  required  skills.  The  nominated 

tasks  are  considered  further  by  the  capability  to 

implement  training  for  the  task  within  the  embedded 
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training  subsystem  of  the  combat  system.  This  phase 

III  evaluation  is  divided  into  six  steps. 

The  first  step  assesses  how  critical  performance  of  a 

task  is  to  mission  accomplishment.  To  perform  this  step, 

subject  matter  experts  are  used  to  place  each  task  in  one 
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of  the  following  categories: 

High  criticality  -  there  is  more  than  a  50  percent 
chance  that  mission  failure  will  occur,  equipment  will  be 
seriously  damaged,  or  personnel  will  be  injured  or  killed, 
if  the  task  or  objective  is  not  performed  correctly. 

Moderate  criticality  -  the  percent  chance  of  mission 
failure  is  between  25  and  50  percent. 

Low  criticality  -  the  percent  chance  of  mission 
failure  is  less  than  25  percent. 


Next,  the  analyst  classifies  each  task  based  on  seven 
categories  of  learning  and  retention  characteristics. 

These  categories  are:  integrated  multiple  skills 
performance;  variable  or  contingency  procedures;  rule  or 
concept  utilization;  invariant  procedures;  basic 
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manipulative  skills;  knowledges;  and  basic  level 
23 

behaviors . 

The  third  step  of  Phase  III  assigns  a  level  of 

perishability  to  the  categories  used  in  the  second  step. 

Since  the  tasks  have  been  assigned  to  one  of  the  seven 

categories,  this  step  identifies  a  general  level  of 

perishability  of  each  task.  Integrated  multiple  skills 

performance  tasks  are  rated  high  perishability.  Variable 

or  contingency  procedure  and  rule  or  concept  utilization 

tasks  are  rated  moderate  perishability.  The  remaining 
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four  categories  of  tasks  are  rated  low  perishability. 

The  fourth  step  nominates  tasks  to  be  considered  for 

embedded  training.  Those  tasks  which  are  rated  as  high  or 

moderately  critical  or  high  or  moderately  perishable  are 

25 

nominated  as  embedded  training  candidates . 

In  the  fifth  step  the  analyst  assesses  if  there  are 
any  unsuitable  tasks  for  embedded  training  and  of  those 
remaining  tasks  which  will  require  large  amounts  of 
resources  to  implement.  This  analysis  uses  a  six  to  seven 
question  algorithm  to  study  methodically  each  nominated 
task.  First,  the  analyst  judges  whether  or  not  the 
stimuli  (e.g.  display  screen)  to  perform  the  task  on  the 
equipment  will  be  feasible.  Second,  the  analyst  decides 
whether  or  not  use  of  embedded  training  to  train  the  task 
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will  result  in  a  safety  hazard  or  damage  to  the 

equipment.  Next  a  decision  is  made  as  to  whether  or  not 

meaningful  performance  measures  can  be  obtained  during  the 

normal  soldier/system  interface  accomplished  during  the 

execution  of  a  task.  For  example,  does  the  task  require 

the  soldier  to  type  in  a  command  or  flip  a  switch  which 

could  be  monitored  by  the  embedded  training  system"? 

Fourth,  the  analyst  decides  whether  or  not  it  is  possible 

to  obtain  performance  feedback  from  the  embedded  training 

system  on  a  particular  task.  Will  there  be  enough 

computer  processing  and  storage  available  either  in  the 

system  or  the  embedded  training  package?  If  not,  can 

observers  assess  performance  validly  on  that  particular 

task"?  The  fifth  question  determines  if  simulation  is 

required  (either  static  or  dynamic)  to  provide  the  stimuli 

necessary  to  perform  the  task.  If  this  simulation  is 

required,  then  the  analyst  answers  the  sixth  question.  Is 

the  simulation  feasible  or  will  the  fidelity  required  in 

the  simulation  put  implementation  of  this  task  at  risk 

either  due  to  cost  or  its  technical  presentation"?  If  the 

answer  is  no  for  any  of  the  questions  then  the  task  is  not 
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considered  acceptable  for  embedded  training. 
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The  final  step  in  phase  III  is  to  conduct  a  review  of 

the  task  nomination  results  to  correct  inconsistencies 

between  higher  level  tasks  and  lower  level  tasks. 

Generally,  if  a  lower  level  task  is  nominated  for  embedded 

training  then  its  higher  level  task  should  also  be 

nominated.  If  this  is  not  the  case  then  the  analyst  must 

examine  the  previous  judgments  to  determine  where  the 
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error  was  made . 

Phase  IV  of  this  procedure  for  developing  embedded 
training  requirements  discusses  how  to  automate  the  data 
obtained  in  phases  I  —  1 1 1  and  presents  a  set  of  report 
formats  which  are  useful  in  the  overall  analysis.  The 
details  and  implementation  of  phase  IV  are  not  relevant  to 
this  project. 

This  four  phased  process  provides  an  effective  method 
to  determine  what  tasks  should  continue  to  be  considered 
for  training  by  the  embedded  training  alternative.  It 
provides  a  rational ,  ordered  process  by  which  to  isolate 
and  study  each  of  the  new  systems’  tasks.  It  excludes 
tasks  which,  for  several  possible  reasons,  cannot  be 
trained  effectively  by  embedded  training.  And  the  method 
recognizes  the  important  role  mission  accomplishment 
(criticality)  and  learning  retention  (perishability)  play 
in  selecting  a  training  alternative. 
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However,  this  screening  process  by  itself  does  not 
provide  enough  available  information  to  the  decisionmaker. 
The  process  provides  no  indication  of  how  well  a  task 
selected  for  embedded  training  might  be  trained  using  a 
different  alternative.  Moreover,  it  does  not  address 
specifically  how  tasks  ought  to  be  trained  if  they  are  not 
selected  as  embedded  training  candidates. 
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CHAPTER  IV.  THE  TRAINING  EFFECTIVENESS  COMPARISON  METHOD. 

This  chapter  develops  the  method  used  to  compare 
embedded  training  to  other  training  alternatives.  Chapter 
I  explained  the  need  for  such  a  method.  Chapter  II 
provided  a  research  summary  of  the  key  methods  available 
today  to  predict  the  effectiveness  of  training  device  or 
program  alternatives.  Chapter  III  provided  the  recently 
developed  method  for  determining  if  a  task  is  appropriate 
for  training  using  the  embedded  training  alternative.  An 
overview  of  the  training  alternative  comparison  method  is 
provided  below: 

Step  1:  Identify  the  training  tasks. 

Step  2:  Determine  the  embedded  training 

Candida  -e  tasks  (ARI  screening  process). 

Step  3:  Identify  the  training  alternatives. 

Step  4:  Structure  the  problem  (Saaty’s 
Analytical  Hierarchy  Process) . 

Step  5:  Obtain  input  data  via  subject  matter 
expert  questionnaire  (training 
effectiveness  criteria  weighting  and 
alternatives  vs  criteria  ratings) . 

Step  6:  Establish  subject  matter  expert  training 
alternative  rankings  (TOPSIS) . 

Step  7:  Aggregate  training  alternative  rankings 
(Borda  function) . 
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A  discussion  of  each  step  is  presented  in  the  subsequent 
paragraphs . 

The  first  step  the  training  analyst  must  accomplish 

is  to  identify  the  training  tasks  necessary  to  operate  the 

new  operational  system  (e.g.  tank,  truck  or  command  and 

control  system) .  This  is  often  accomplished  by  agencies 

outside  the  office  directly  responsible  for  the  training 

analysis  accomplished  during  the  Preliminary  Training 

Effectiveness  Analysis.  If  the  task  list  for  the  new 

system  is  not  provided  then  the  analyst  has  several 

alternatives.  Phase  I  of  the  ARI  screening  process 

(described  in  Chapter  III)  can  be  used  to  develop  the  task 

list.  Another,  less  desirable  method,  is  to  develop  a 

task  list  by  identifying  a  similar  existing  system  and 
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developing  an  analogous  task  list  for  the  new  system. 

Next,  the  analyst  determines  the  tasks  which  are 
appropriate  for  embedded  training  using  the  ARI  screening 
process.  This  process  was  described  in  detail  in  Chapter 
III.  Phase  III  of  this  process  provides  the  actual 
screening  method.  Use  of  this  process  will  be 
demonstrated  in  Chapter  V.  This  screening  step  in  the 
training  effectiveness  comparison  method  is  important. 
Since  embedded  training  is  a  new  training  alternative 
little  is  know  about  its  applicability  to  a  certain  task 
for  a  certain  developing  system.  The  screening  step  helps 


35 


to  ensure  that  a  task  which  is  not  suited  for  embedded 

training  does  not  include  the  embedded  training 

alternative  during  the  later  comparison  steps.  The 

counterpart  methods  for  determining  what  present  day 

training  alternatives  are  appropriate  for  the  task  under 

consideration  are  discussed  below  (Step  3) . 

In  step  3,  the  analyst  identifies  the  training 

alternatives  to  compare  against  embedded  training. 

Training  alternatives  are  the  instructional  methods  used 

to  train  the  particular  task  under  consideration.  They 

can  range  from  the  traditional  classroom  setting  through  a 

full  fidelity  stand  alone  simulator  (e.g.  flight 

simulator).  A  discussion  of  training  equipment, 

materials,  and  methods  (which  form  alternatives)  is 

provided  by  the  Training  Efficiency  Estimation  Model 

(TEEM)  found  in  Chapter  II  and  the  Technique  for  Choosing 

Cost  Effective  Instructional  Delivery  Systems 
29 

(TECEP) .  The  analyst  may  chose  any  number  of 

alternatives  which  are  appropriate  for  a  particular  task 

under  consideration.  However,  the  total  number  of 
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alternatives  should  remain  under  ten.  Beyond  this 
point  several  problems  emerge.  First,  the  size  of  the 
problem  causes  computational  difficulty.  Second,  the  size 
of  the  data  collection  instrument  becomes  overwhelming. 

And  finally,  when  large  numbers  of  training  alternatives 
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are  used  it  becomes  difficult  to  distinguish  (and  present 
results  to  decisionmakers)  between  alternatives.  Since 
the  purpose  of  this  project  is  to  provide  additional 
information  about  tasks  considered  appropriate  for 
embedded  training,  embedded  training  will  normally  be 
considered  as  a  training  alternative.  However,  this 
method  can  also  be  used  to  identify  the  most  appropriate 
training  alternative  for  those  tasks  not  considered 
appropriate  for  embedded  training. 

Once  the  training  alternatives  have  been  identified 
it  is  now  possible  to  structure  the  problem  (step  4) .  In 
our  case,  we  seek  a  model  which  will  permit  us  to 
evaluate,  for  a  particular  task,  several  training 
alternatives  under  a  set  of  training  effectiveness 
criteria  with  the  ultimate  goal  of  identifying  the  most 
effective  training  alternative.  Saaty’s  Analytical 
Hierarchy  Process  (AHP)  provides  such  a  model . 

The  general  form  of  the  hierarchial  structure  used  in 
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the  Analytical  Hierarchy  Process  is: 

FIRST  LEVEL  GOAL  OR  OBJECTIVE 

SECOND  LEVEL  CRITERIA  (1  to  9) 

THIRD  through  ADDITIONAL  CRITERIA  RELATED 

n-1  LEVEL  TO  THE  CRITERIA  IN  THE 

PREVIOUS  LEVEL 

FINAL  LEVEL  ALTERNATIVES 

(nth  LEVEL) 
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Saaty's  model  uses  this  hierarchial  approach  to  break  down 

a  complex  problem  into  pieces  that  can  be  more  adequately 

evaluated.  It  provides  an  evaluation  process  which  gives 

a  decisionmaker  the  opportunity  to  choose,  based  upon 

certain  criteria,  an  alternative  which  provides  the  best 

solution  to  attain  a  certain  objective.  The  hierarchy  is 

described  by  levels.  The  first  level  is  the  goal  or 

objective  of  the  decisionmaking  problem.  The  second  and 

subsequent  intermediate  levels  (n-1)  contain  the  criteria 

used  to  identify  the  alternatives.  The  final  level 

consists  of  the  alternatives  used  to  attain  the  objective. 

Our  application  of  the  hierarchial  structure  fixes 

the  first  and  second  levels  of  the  structure.  The  first 

level  is  our  objective:  identifying  the  most  effective 

training  alternative  for  each  task.  The  second  level  is 

comprised  of  the  training  effectiveness  criteria  used  for 

evaluating  each  of  the  alternatives.  Six  criteria  have 

been  identified  from  the  research  summarized  in  Chapter 

II.  They  are  stimuli  presentation,  response  presentation, 
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feedback  presentation,  task  commonality,  physical 

33 

similarity,  and  training  time.  These  criteria  are 

discussed  in  the  next  six  paragraphs. 

Stimuli  presentation  (C  )  provides  the  analyst  with 

1 

an  evaluation  criterion  used  to  judge  the  potential  that 
alternatives  have  to  present  information  for  the 
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particular  task  under  consideration .  Based  upon  the 
analyst’s  (or  subject  matter  expert)  knowledge  of  the 
gunner’s  station  of  the  Future  Armored  Combat  System  and 
his  knowledge  of  predecessor  systems,  he  determines  what 
type  of  stimuli  is  required  to  perform  the  task  under 
consideration.  The  types  of  stimuli  are  visual  cues, 
audio  cues,  tactical  cues,  external  stimulus  motion  cues, 
internal  stimulus  motion  cues,  olfactory  cues,  and 
gustatory  cues.  The  analyst  then  judges  the  extent  to 
which  a  particular  training  alternative  can  provide  the 
required  stimuli  to  perform  the  task. 

The  response  presentation  criterion  is  used  to 
evaluate  the  extent  to  which  the  training  alternatives 
permit  a  response  to  the  information  presented  to  the 
trainee  or  crewmember  for  the  particular  task  under 
review.  In  this  case,  the  analyst  first  determines  what 
type  of  response  is  required  of  the  operator  to  perform 
the  task  under  consideration.  This  response  can  take  one 
of  the  following  forms:  verbal,  written,  manipulative 
act,  tracking,  or  procedural  performance.  The  analyst 
then  judges  the  extent  to  which  a  particular  training 
alternative  can  accept  that  type  of  response. 

The  feedback  presentation  criterion  is  used  to 
evaluate  the  extent  to  which  the  training  alternatives  can 
provide  feedback  to  the  operator.  Again,  the  analyst 
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first  determines  the  type  of  feedback  required  to 
reinforce  correct  performance  or  correct  faulty 
performance  on  a  particular  task.  This  determination  is 
made  based  upon  the  analyst’s  knowledge  of:  the  gunner’s 
station  in  the  Future  Armored  Combat  System,  the  gunner’s 
station  in  predecessor  systems,  and  the  four  training 
alternatives.  Feedback  presentation  takes  one  of  the 
following  forms:  visual,  aural,  written,  face-to-face 
communication,  indirect  communication,  tactile, 
kinesthetic,  olfactory,  and  gustatory.  The  analyst  then 
judges  the  extent  to  which  a  particular  training 
alternative  can  provide  the  required  feedback. 

The  task  commonality  criterion  is  used  to  evaluate 
the  extent  to  which  the  training  alternatives  can  train 
the  subtasks  (basic  steps)  of  a  particular  task.  The 
subtasks  of  a  particular  task  are  either  provided  to  the 
analyst  or  the  analyst  makes  an  estimate  of  what  the 
subtasks  might  be  based  upon  his  knowledge  of  the 
developing  system  and  its  predecessor  systems.  The 
analyst  then  estimates  the  percentage  of  subtasks  a 
particular  training  alternative  would  be  able  to  train. 

The  physical  similarity  criterion  is  used  to  evaluate 
the  extent  to  which  the  representation  of  the  displays  and 
controls  of  the  four  training  alternatives  are  similar  to 
the  displays  and  controls  of  the  Future  Armored  Combat 
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System  for  a  particular  task  under  consideration. 


Since 


little  is  known  of  the  developing  system  at  this  early 
stage  of  its  development  cycle,  the  analyst  must  determine 
if  the  particular  training  alternative  has  the  potential 
to  provide  display  and  control  physical  similarity  to  what 
likely  will  be  found  in  the  gunner’s  station  of  the  Future 
Armored  Combat  System. 

The  final  criterion  used  to  evaluate  the 
effectiveness  of  a  training  alternative  for  a  particular 
task  is  training  time.  This  criteria  is  used  to  evaluate 
the  length  of  training  time  required  to  train  a  particular 
task  using  one  of  the  four  training  alternatives.  It  is 
not  measured  in  absolute  time  (something  a  later  field 
test  would  establish)  but  rather  reflects  a  comparison 
among  alternatives. 

Therefore,  the  general  hierarchial  structure  becomes: 


FIRST  !  TRAINING  EFFECTIVENESS  ! 

LEVEL  I  OF  A  PROPOSED  TRAINING  SUBSYSTEM!  OBJECTIVE 


! STIMULI  !  ! FEEDBACK  !  ! PHYSICAL  ! 

SECOND  ! PRESENTATION!  ^PRESENTATION!  iSIMILARITYi 
LEVEL 

_ CRITERIA 

iRESPONSi  ~  !  ! TASK  !  i TRAINING! 

! PRESENTATION!  ! COMMONALITY !  ! TIME  ! 


! ALTERNATIVE  1!  !  ALTERNATIVE^  ! 

THIRD 

LEVEL  _  ALTERNATIVES 

TlMBEDDED! 

! TRAINING ! 
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Typical  training  alternatives  will  be  incorporated  when 

the  model  is  demonstrated  using  a  sample  system. 

Step  5  continues  the  Analytical  Hierarchy  Process. 

In  this  step,  subject  matter  experts  (a  single  training 

analyst  or  decisionmaker  could  also  perform  this  step) 

establish  a  relationship  between  the  criteria  and  the 
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objective  and  the  alternatives  and  the  criteria. 

First,  the  subject  matter  expert  establishes  the 
contribution  of  a  particular  criterion  to  achieve  the 
objective.  This  is  done  through  a  pairwise  comparison 
process  to  establish  a  numerical  weight  to  each  of  the 
criteria.  In  this  case,  the  relationship  between  criteria 
and  objective  can  be  expressed  in  matrix  form: 


Effective  Training 
Alternative 

SP  :  RP  :  FP 

TC 

ps  :  tt  : 

SP 

l  :  : 

i  i 

i  » 

RP 

:  l  : 

i  i 

<  i 

FP 

:  :  l 

i  i 

i  i 

TC 

<  i 

i  i 

1 

i  i 

i  i 

PS 

i  i 

i  i 

l  i  ; 

TT 

i  i 

i  i 

:  l  : 

FP=  Feedback  Presentation  SP=  Stimulus  Presentation 

PS=  Physical  Similarity  TC=  Task  Commonality 

RP=  Response  Presentation  TT=  Training  Time 

It  turns  out  that  the  pairwise  comparison  matrix  is 

reciprocal.  This  means  that  the  comparison  value  for 

SP:RP  is  equal  to  1/ (comparison  value  for  RP:SP)  or 

a  =l/a  .  For  this  reason  it  is  only  necessary  to 
j  i  i  j 

make  the  comparisons  for  the  top  half  of  the  matrix. 

-  4 


In  a 


general  sense,  n(n-l)/2  establishes  the  necessary  number 


of  comparisons.  In  our  case  15  comparisons  [6(6-l)/2]  are 

required  out  of  a  possible  30. 

The  subject  matter  expert  begins  by  comparing  the 

contribution  stimuli  presentation  has  to  training 

effectiveness  versus  the  contribution  response 

presentation  has  to  training  effectiveness  using  an 

established  comparison  scale.  This  process  is  repeated 

through  the  15  pairs  in  the  top  half  of  the  matrix.  The 
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comparison  scale  is: 

THE  PAIRWISE  COMPARISON  SCALE 


Intensity  of 

Importance  Definition 

1  Equal  importance  of 

both  elements 


3  Weak  importance  of 

one  element  over 
the  other 

5  Essential  or  strong 

importance  of  one 
element  over  the 
other 

7  Demonstrated 

importance  of  one 
element  over  the 
other 

9  Absolute  importance 

of  one  element  over 
the  other 


Explanation 

Two  elements  contribute 
equally  to  training 
ef  f ectiveness 

Experience  and  judgment 
slightly  favor  one 
criteria  over  the  other 

Experience  and  judgment 
strongly  favor  one 
criteria  over  the  other 


A  criteria  is  strongly 
favored  and  its  dominance 
is  demonstrated  in 
practice 

The  evidence  favoring  one 
criteria  over  another  is 
of  the  highest  possible 
order  of  affirmation 
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2,4,  Intermediate  values  Compromise  is  needed 

6,8  between  two  adjacent  between  two  judgments 

j  udgments 

Reciprocals 

If  the  second  item  of  comparison  is  the  stronger 
contributor  to  training  effectiveness  then  the 
score  given  in  the  comparison  rating  is  the 
reciprocal  value  of  the  appropriate  rating  in  the 
scale  above  (e.g.  1/3 , 1/5 , 1/6) . 

A  few  comments  about  this  scale  and  the  comparison 

process  are  appropriate  at  this  point.  If  we  knew  the 

physical  relationship  of  the  criteria  with  respect  to  each 

training  alternative  (e.g.  training  time  for  embedded 

training  for  Task  A  was  27  minutes)  then  we  could  proceed 

directly  to  normalize  the  scores  and  solve  the  problem  by 

one  of  several  multiple  attribute  decisionmaking  models. 

Moreover,  the  rankings  established  by  these  data  would 

provide  more  real  information  about  how  much  better  one 

alternative  was  over  another.  Unfortunately,  this  type  of 

concrete  information  is  not  available  at  this  early  stage 

of  the  development  cycle.  So  what  must  be  done  is  to 

establish  a  way  of  capturing  experience,  knowledge,  and 

predictions  of  the  future  to  decide  which  task  is  best 

trained  by  which  training  alternative.  Saaty’s  approach 
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can  be  used  to  do  this.  The  pairwise  comparison 
process  provides  a  logical,  step  by  step  process  to  help 
ensure  our  decisions  about  the  effect  each  level  has  on 
the  preceding  level  (e.g.  criteria  on  the  objective. 
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training  alternatives  on  the  criteria)  remains 
consistent.  In  other  words,  if  we  say  A  is  twice  as 
important  as  B  and  B  is  three  times  as  important  as  C ,  we 
do  not  say  that  C  is  twice  as  important  as  A.  Saaty ’ s 
scale  also  helps  to  ensure  that  responses  are  consistent 
by  providing  a  standardized  way  of  assigning  relative 
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importance  for  each  of  the  pairs  under  consideration. 

The  alternative  would  be  to  assign  arbitrary  numbers  - 
numbers  which  might  well  lose  their  original  meaning  after 
the  numbers  of  comparisons  increase. 

Once  the  necessary  15  comparisons  are  established 

then  they  are  entered  into  the  criteria  matrix  shown 

earlier.  The  reciprocal  values  of  the  top  half  of  the 

matrix  form  the  entries  for  the  bottom  half.  The  weights 

are  then  estimated  using  the  eigenvector  method  (this  is 

possible  sin9«  all  matrices  completed  our  hierarchial 

structure  are  square) .  The  eigenvector  method  sets  the 

determinant  of  A  -  p.1  =  0  (A  is  the  pairwise  comparison 

matrix)  and  solves  for  the  largest  eigenvalue  (which  is 

the  largest  root  of  this  characteristic  equation)  of  A 

(i.e.  (L  max).  The  value  for  7-  max  is  then  used  in  the 

matrix  equation  (A  -^maxl)  w  =  0  to  obtain  the 

eigenvector  w  or  weights.  Many  computer  programs  are 

available  which  compute  the  eigenvalue  and  eigenvector  of 
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a  square  matrix. 
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The  largest  eigenvalue  also  is  used  to  measure 

consistency  of  the  comparisons  in  a  particular  comparison 

matrix.  A  measure  of  the  'goodness'  of  these  judgments  is 

called  the  consistency  index.  This  index  measures  the 

difference  between  our  matrix  of  pairwise  comparisons  and 

a  consistent  matrix.  It  is  the  measure  of  deviation  of 

the  largest  eigenvalue  (2  max)  of  our  matrix  from  the 

largest  eigenvalue  (n)  of  a  consistent  matrix.  The 

consistency  index  is  given  by  the  formula:  (/L  max  -  n) / (n 

-  1).  According  to  Saaty:  "In  general,  if  the  number  is 

less  than  0.1,  we  may  be  satisfied  with  our 
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Judgments.’  Saaty  goes  beyond  this  generalization  and 
develops  a  consistency  ratio.  This  ratio  provides  a 

comparison  of  the  consistency  index  generated  by  the 
subject  matter  expert's  pairwise  comparisons  and  a  random 

index.  To  establish  a  random  index,  Saaty  and  several 
colleagues  randomly  generated  forced  reciprocal  matrices 
of  rank  order  1  through  11  using  a  sample  size  of  500. 

The  500  consistency  indices  for  each  of  the  11  matrix 
groups  were  then  averaged  to  obtain  a  random  index  for 
each  rank  order  group.  We  are  interested  in  the  random 
index  for  a  matrix  of  rank  six  -  the  criteria  versus  goal 
matrix  -  and  a  matrix  of  rank  four  -  the  training 
alternatives  versus  criterion  matrix.  The  random  index 
thus  established  for  a  matrix  of  rank  six  is  1.24  and  a 
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matrix  of  rank  four  is  0.90.  The  consistency  ratio  is 
found  by  dividing  the  consistency  index  by  the  random 

index  for  the  same  order  matrix.  Saaty  considers  a 
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consistency  ratio  of  . 1  or  less  acceptable.  Thus  for 
a  matrix  of  rank  six,  Saaty  recommends  that  the 
consistency  index  be  .124  or  less  (CI/RI  =  . 1  or  CI/1.24  = 

.1  or  Cl  =  .124)  and  for  a  matrix  of  rank  four  the 
consistency  index  be  .09  or  less  (CI/RI  =  . 1  or  Cl/. 9  =  .1 
or  Cl  =  .09) .  Use  of  this  number  gives  the  decisionmaker 
an  idea  of  the  extent  to  which  a  subject  matter  expert  has 
responded  to  the  pairwise  comparison  in  a  random  fashion. 
Failure  to  be  below  the  0.1  level  does  not  mandate 
elimination  of  the  data  set.  However,  large  values  of  the 
consistency  index  suggest  that  the  particular  comparison 
should  be  retried. 

Saaty  does  provide  several  suggestions  for  attacking 
the  problem  of  identifying  which  part  of  the  comparison 
matrix  to  retry.  One  way  is  to  form  the  matrix  of 


priority  ratios  w  /w  ,  determine  the  matrix  of 

i  J 

absolute  differences  C/a  -  (w  /w  ) I ] ,  and  revise 

i  j  i  J 

the  Judgment  on  the  element (s)  or  row  sums  with  the 


largest  differences, 
mean  square  deviation 


Another  way  is  to  determine 

1  1/n  >  [ (a  )  -  (w  /w  ) ] 
A?>  iJ  1  J 


for  each  of  the  i  rows  and  remake  the  comparisons 


the  root 


i  n  the 


row  with  the  largest  value.  Overuse  of  these  procedures 
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can  be  counterproductive.  Saaty’a  caution  is:  "We 

caution  against  excessive  use  of  this  process  of  forcing 

the  values  of  Judgments  to  improve  consistency.  It 

distorts  the  answer.  One  would  rather  have  naturally 

4  1 

improved  judgments  arising  from  expertise.'  This  is 
true  in  our  case.  Unless  time  is  extremely  limited, 
matrices  with  a  high  consistency  index  should  be 
regenerated . 

The  pairwise  comparison  process  is  repeated  to 
establish  the  effect  the  alternatives  have  on  the 
criteria.  In  our  application  of  the  hierarchial  structure 
a  pairwise  comparison  matrix  is  formed  for  each  criteria 
(C)  with  rows  and  columns  formed  by  the  number  of 
alternatives.  The  general  criterion  matrix  is: 


A1 

1 

A2 

1 

An 

• 

i  : 

The  1-9  rating  scale  is  again  used  to  complete  the  entries 

(a  )  in  the  six  criterion  matrices  (SP , RP , FP , TC , PS , TT) 
i  J 

for  each  matrix  the  weights  (ranking  of  alternatives)  are 
established  using  the  eigenvector  method. 

The  sixth  step  establishes  the  subject  matter 
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expert’s  ranking  of  training  alternatives  based  upon  the 

weights  established  in  step  5.  In  our  case  (a  model  with 

three  levels)  Saaty's  solution  method  is  essentially  the 
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Simple  Additive  Weighting  Method.  Step  5  has 
established  two  vectors  of  weights: 


LEVEL  2:  Criteria: 
Goal  of  Training 
Ef  f  ectiveness 


LEVEL  3:  Training  Alternatives 
Cr i ter ia 


TE 

SP 

RP 

FP 

TC 

PS 

TT 

SP 

r  wll-1 

A1 

r  w  i  i 

wl  2 

wl3 

wl  4 

wl  5 

wie 

RP 

;  w2 1 

A2 

:  w2i 

w2  2 

w2  3 

w24 

w2  5 

w26 

=  FP 

:  w3 1 

B3  * 

i 

. 

r 

TC 

:  w4 1 

. 

i 

. 

. 

. 

. 

PS 

:  w5 1 

. 

i 

. 

. 

TT 

^  w6! 

An 

L  wnl 

wn2 

wn3 

wn4 

wn5 

wn6 

In  Saaty's  method  these  two  matrices  are  multiplied  to 
obtain  the  composite  ranking  of  training  alternatives: 


A 1  rwll"! 


W 


B2B3 


A2 


w2  1 


An  !_  wn  1  J 

The  alternatives  are  ranked  by  their  composite  numerical 
value  (i.e.  the  highest  wnl  is  the  most  effective  training 
alternative  when  compared  to  the  other  training 
alternatives) . 

Saaty’s  solution  approach  is  simple  and  frequently 
quite  sufficienw.  Occasionally,  however,  the  composite 
vector  will  produce  alternative  weights  which  are  equal. 
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To  avoid  this  tie  situation  TOPSIS  was  selected  over 


Saaty ' s  solution  approach  to  establish  the  individual 

subject  mater  expert  training  alternative  rankings. 

TOPSIS  is  '[also]  simple  and  yields  an  indisputable 
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preference  order  of  solution.'  TOPSIS  achieves  this 
result  by  working  under  the  principle  that  the  selected 
alternative  should  have  two  properties.  First,  it  should 
be  closest  to  the  best  possible  solution  (called  the  ideal 
solution) .  Second,  it  should  also  be  farthest  from  the 
worst  possible  solution  (called  the 

negative- idealsolut ion) .  This  concept  can  be  illustrated 
graphical ly : 


STIMULUS  PRESENTATION 

If  we  look  at  the  two  dimensional  comparison  of  the 

effect  of  stimulus  presentation  and  response  presentation 

on  several  alternatives  (A  )  we  can  see  that  A  has 

n  1 
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jt 

the  shortest  distance  to  ths  ideal  solution  (A  )  and  the 

t 

t 

negative-ideal  solution  (A  ).  However,  A  is  the  same 

2 

distance  from  the  ideal  solution  as  A  but  is  farther 

1 

from  the  negative- ideal  solution.  In  this  case,  A  is 

2 

selected  because  it  has  the  best  of  both  properties. 

TOPSIS  actually  produces  a  number  called  the  relative 

closeness  to  the  ideal  solution  and  it  is  this  number  that 

is  used  to  rank  order  the  alternatives  (vice  the  weights 

used  in  the  composite  vector  of  Saaty's  solution 

approach).  TOPSIS  is  discussed  in  detail  in  Appendix  C. 

The  final  step  (step  7)  is  to  aggregate  the  input 

from  several  decisionmakers.  The  intent  of  this  step  is 

to  provide  the  decisionmaker  with  a  summary  alternative 

ranking  recommendation  that  accounts  for  the  subject 

matter  experts'  experience,  knowledge,  and  judgment. 

Hwang  and  Lin  present  a  comprehensive  survey  of  social 
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choice  methods  of  aggregating  rank  order  input.  Some 
examples  are  the  functions  of  Condorcet,  Borda ,  Copeland, 
Nanson,  Dodgson,  and  Kemeny .  The  Borda  function  was 
selected  for  our  application  because  it  is  computational ly 
simple  and  it  gives  positive  credit  to  an  alternative  any 
time  it  is  selected  over  another. 

This  positive  credit  is  a  key  advantage  of  the  Borda 
technique.  Since  data  available  at  this  early  point  in 
the  developmental  cycle  actually  reflect  judgment  vice 
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performance  data  obtained  from  a  teat,  it  ia  to  our 
advantage  to  uae  aa  much  information  aa  poaaible.  We  want 
to  know  not  only  which  alternative  our  subject  matter 
experts ’  Judgment  and  experience  determine  ia  the  aimple 
majority  winner  but  alao  to  what  extent  the  other 
alternativea  are  favored  over  each  other.  The  Borda 
technique  helpa  ua  do  that  because  it  accounts  for  each 
choice  of  one  alternative  over  another  -  be  it  between  the 
firat  and  aecond  place  alternative  or  the  third  and  fourth 

place  alternative.  Thia  ia  an  improvement  over  methoda 
which  uae  the  aimple  majority  vote,  penalize  the 
alternative  for  losses ,  or  attempt  to  gain  a  conaenaua 
ranking.  It  ia  an  improvement  becauae  we  have  captured 
each  piece  of  Judgment  and  experience  which  haa 
contributed  to  the  ranking  of  one  alternative  over  another 
and  then  aggregated  that  information  across  all  the 
subject  matter  experts  for  a  final  alternative  ranking. 

The  Borda  function  aggregation  technique  awards  a 
different  number  of  points  to  each  alternative  depending 
upon  whether  it  was  ranked  firat,  second,  third  and  so  on. 
The  points  awarded  are  established  by  the  rule:  m- 1 , 
m-2,...,  1 ,  0  to  the  first,  second,  ...,  last  place 

alternative  where  m  is  the  number  of  al ternati vea .  The 
Borda  score  for  a  particular  alternative  is  then 
established  by  summing  the  points  awarded  for  that 
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alternative.  The  alternative  with  the  highest  number  of 
points  is  declared  the  winner  and  the  remaining 

alternatives  are  ranked  in  decreasing  order  of  their  total 
45 

score . 

An  example  will  illustrate  the  application  of  this 
technique.  Let  us  assume  we  have  four  subject  matter 
experts  (SME)  whose  individual  alternative  rankings 
determined  by  TOPSIS  were: 


SME 

1 

SME 

2 

SME 

3  SME 

4 

R  1 

A3 

(3) 

A4 

(3) 

A1 

(3)  A2 

(3) 

A  2 

A1 

(2) 

A1 

(2) 

A2 

(2)  A3 

(2) 

N  3 

A4 

(1) 

A3 

(1) 

A3 

(1)  A1 

(1) 

K  4 

A2 

(0) 

A2 

(0) 

A4 

(0)  A4 

(0) 

The  number 

in 

parenthesis 

i  s 

the 

Borda  points  awarded 

based 

on  the  application 

of 

the 

rule 

:  (4  -  1)  = 

3 ,  f irst 

place ; 

(4 

-  2) 

*  2 

,  second  place 

;  (4-3)  *  1  , 

third 

place ; 

and 

(4 

-  /  , 

=  0,  last 

place . 

The  Borda 

score  for 

each  alternative  is: 


A3  =  3 

+ 

1 

+ 

1 

+ 

2 

A 1  =  2 

+ 

2 

+ 

3 

+ 

1 

A4  =  1 

+ 

3 

+ 

0 

+ 

0 

A2  =  0 

+ 

0 

+ 

2 

♦ 

3 

The  final  alternative  ranking  for  the  four  subject  matter 
experts  is  :  A1  (8)  ,  A3  (7)  ,  A2  (4)  ,  A4  (4)  . 

If  ties  exist  in  the  individual  rankings  then  the 
place  finish  is  established  by  the  lowest  position  had 
there  been  no  tie.  If  there  are  three  ties  for  first 
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place  then  the  points  awarded  to  all  three  tied 


alternatives  Is  equal  to  the  third  place  rule  or  m  -  3 . 

For  example,  a  subject  matter  expert  has  the  following 
ranking:  first  (tie)  -  A6 ,  A4 ,  A2 ;  second  -  A1 ;  third  - 

A3,  A5 .  His  ranking  without  ties  might  be:  first  -  A6 , 
second  -  A4 ,  third  -  A2 ,  fourth  -  A1 ,  fifth  -  A3,  sixth  - 
AS.  Since  the  lowest  ranking  of  the  first  place 
alternatives  is  third,  alternatives  A6 ,  A4 ,  and  A2  are 
each  awarded  6  -  3  or  3  points.  The  lowest  ranking  of  the 
third  place  alternatives  is  sixth  so  A3  and  AS  are  awarded 
0  points. 

It  is  interesting  to  note  that  the  final  Borda  score 
for  an  alternative  reflects  the  number  of  times  the  four 

subject  matter  experts  have  picked  a  particular 
alternative  over  the  other  three  alternatives.  For 

example,  SME  #  1  has  picked  A3  over  3  other  alternatives, 
SME  #  2  and  #  3  have  picked  A3  over  one  other  alternative, 
and  SME  #  4  has  picked  A3  over  2  other  alternatives.  It 
is  the  knowledge  and  experience  based  Judgment  of  the 
subject  matter  experts  that  has  determined  that  A3  would 
be  picked  over  other  alternatives  seven  times.  This  more 
precise  information  of  how  an  alternative  fared  against 
the  other  alternatives  is  what  makes  the  Borda  technique 
so  valuable. 
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CHAPTER  V.  THE  FUTURE  ARMORED  COMBAT  SYSTEM:  AN 
APPLICATION  OF  THE  SCREENING  PROCESS. 

This  chapter  evaluates  the  Future  Armored  Combat 
System  tasks  using  the  Roth,  Sullivan,  and  Ditzian  four 
phased  process  identified  in  Chapter  III.  Phase  I,  which 
defines  the  tasks  that  must  be  performed  to  operate  the 
new  piece  of  equipment,  has  already  been  accomplished  (it 
will  no  doubt  be  updated  as  the  system  evolves  through  its 
development  cycle)  by  training  analysts  of  the  Training 
Division,  U.S.  Army  Armor  School  (1987-  see  Appendix  A) . 

To  provide  a  reasonable  scope  for  the  development  of  the 
training  effectiveness  methodology  in  the  next  chapter 
only  the  individual  tasks  for  the  gunner’s  station  will  be 
examined  as  potential  embedded  training  candidates. 

As  discussed  in  chapter  III,  phase  II  is  an  optional 
phase  performed  when  specific  data  required  for  detailed 
task  analysis  is  available.  Although  materiel  developers 
presently  know  what  capabilities  will  be  required  of  the 
Future  Armored  Combat  System  and  have  a  general  idea  of 
what  technologies  will  be  used  to  accomplish  those 
capabilities  they  do  not  yet  know  exactly  how  the  operator 
will  perform  each  task  and  what  stimuli  will  be  provided 
by  the  equipment  during  task  performance.  Phase  II  will 
therefore  not  be  performed. 
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The  third  phase  was  performed.  It  is  this  phase 
which  identifies  the  candidate  tasks  for  embedded 
training.  The  results  of  this  six  step  process  are 
provided  in  Table  1.  Tasks  selected  are  identified  with 
an  asterisk. 
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TABLE  1.  EMBEDDED  TRAINING  CANDIDATE  TASK  ANALYSIS  FOR 
THE  GUNNER’S  POSITION  ON  THE  FUTURE  ARMORED  COMBAT  SYSTEM. 


Task 

Critical i ty 

Learning  & 
Retention 
Classi f ication 

Recover  by  similar  vehicle 

High 

Contingency 

Procedure 

Remove/ Ins tal 1  Track  Blk(s) 

Moderate 

Invariant 

Procedure 

"Perform  Before  Operations 
Checks  and  Services 

Moderate 

Invariant 

Procedure 

Perform  During  Operations 
Checks  and  Services 

High 

Invariant 

Procedure 

Perform  After  Operations 
Checks  and  Services 

Low 

Invariant 

Procedure 

Perform  Gunner’s  Prep  to 

Fire  Checks  &  Services 

High 

Invariant 

Procedure 

Perform  Gunner’s  After 

Fire  Checks  &  Services 

Low 

Invariant 

Procedure 

Prepare  Gunner's  Station 
for  Operation 

High 

Invariant 

Procedure 

Secure  Gunner’s  Station 

Moderate 

Invariant 

Procedure 

"Troubleshoot  the  Fire 
Control  System 

Low 

Contingency 

Procedure 

Prepare  Gunner ’ s  Weapon 
for  Travel 

Low 

Invariant 

Procedure 

"Evacuate  a  Wounded 

Crewman 

Moderate 

Multi  Skills 
Performance 

Extinguish  a  Fire 

High 

Rule 

Util ization 

"Replace  a  Thrown  Track 

High 

Rul  e 

Util ization 

Operate  the  Gas 

Particulate  System 

Hi  gh 

Invariant 

Procedure 

Boresight  and  System 

Cal ibrate 

High 

Invariant 

Procedure 

Perform  Manual  Extract¬ 
ion  Main  Gun  Round 

High 

Invariant 

Procedure 

Prepare  for  Powerpack 

Removal 

High 

Invariant 

Procedure 

Perform  Operator  Before 

C&S  NBC  System 

High 

Invariant 

Procedure 

Inspect  Main  Gun  Ammo 
for  Serviceability 

Moderate 

Rule 

Util ization 

Stow  Ammunition 

Moderate 

Invariant 

Procedure 
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Perish¬ 

ability 

Embedded 
Task  ? 

Implementation  Feasibility 
and  Approaches 

On  Equip  Task 
Related  Stimuli 

Safety 

Problem 

Can  Perf  Meas 
Be  Defined 

Moderate 

Yes 

Feasible 

No 

No 

Low 

Yes 

Feasible 

No 

No 

Low 

Yes 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

Yes 

Low 

No 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

Yes 

Low 

No 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

Yes 

Moderate 

Yes 

Feasible 

No 

Yes 

Low 

No 

Feasible 

No 

No 

High 

Yes 

Feasible 

No 

No 

Moderate 

Yes 

Feasible 

No 

Yes 

Moderate 

Yes 

Feasible 

No 

No 

Low 

Yes 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

Yes 

Low 

Yes 

Feasible 

No 

No 

Low 

Yes 

Feasible 

No 

No 

Low 

Yes 

Feasible 

No 

No 

Moderate 

Yes 

Feasible 

No 

No 

Low 


Yes 


Feasible 


No 


No 


Implementation  Feasibility 
and  Approaches 

Final 

ET 

Will  Assess 
Be  Positive 

Can  Personnel 
Assess  Perf 

Is  Sim 
Needed 

Is  Sim 
Feas i ble 

Assess¬ 

ment 

No 

Yes 

No 

NA 

Yes  . 
line 

Off 

asmt 

No 

Yes 

No 

NA 

Yes  . 
line 

Off 

asmt 

Yes 

Yes 

No 

NA 

Yes 

Yes 

Yes 

No 

NA 

Yes 

Yes 

Yes 

No 

NA 

No 

Yes 

Yes 

No 

NA 

Yes 

Yes 

Yes 

No 

NA 

No 

Yes 

Yes 

No 

NA 

Yes 

Yes 
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The  phase  III  analysis  eliminated  three  of  the 
thirty-seven  tasks  under  review.  Specifically,  perform 
after  operations  checks  and  services,  perform  gunner’s 
after  fire  checks  and  services,  and  prepare  gunner’s 
weapon  for  travel  were  eliminated  because  they  were  rated 
low  in  both  the  criticality  and  perishability  categories. 
Of  the  remaining  thirty-four,  twenty-three  were  nominated 
as  embedded  training  candidates  requiring  an  off  line 
training  assessment  capability.  It  is  unlikely  that  the 
equipment  found  in  the  gunner’s  station  will  have  the 
ability  to  keep  track  of  the  procedure  the  soldier  would 
use  to  perform  these  twenty-three  tasks.  Eleven  tasks 
were  found  to  be  appropriate  for  full  embedded  training. 
That  is,  it  is  likely  that  training,  evaluation,  and 
feedback  can  be  provided  by  the  equipment  at  the  gunner’s 
station  for  these  eleven  tasks. 

Since  trying  to  apply  all  tasks  nominated  for  embedded 
training  (34)  to  the  demonstration  of  the  training 
effectiveness  methodology  would  complicate  dramatically 
the  intent  of  this  project,  I  will  restrict  further 
analysis  to  the  following  eight  tasks: 

-  Perform  before  operations  checks  and  services 

-  Troubleshoot  the  fire  control  system 

-  Evacuate  a  wounded  crewman 
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-  Replace  a  thrown  track 

-  Engage  targets  with  the  main  gun  from  the  gunner’s 
station 

-  Adjust  crew  compartment  pressurization 

-  Zero  the  machinegun 

-  Engage  targets  with  the  coax  from  the  gunner’s 
station 

This  task  selection  provides  a  mix  of  full  embedded 
training  tasks,  embedded  training  tasks  with  off  line 
assessment,  and  a  mixture  of  the  other  analysis  categories 
(e.g.  criticality,  learning  and  retention  classification). 
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CHAPTER  VI.  SETTING  UP  THE  PROBLEM:  ALTERNATIVES, 

PROBLEM  STRUCTURE,  INPUT  DATA  (STEPS  3,4,5). 

Now  that  we  have  established  training  candidate  tasks 
(previous  chapter)  we  want  to  investigate,  step  3  of  the 
training  effectiveness  comparison  method  is  to  identify 
other  training  alternatives  to  compare  against  embedded 
training.  For  this  demonstration  three  additional 
training  alternatives  were  chosen.  They  are:  classroom 
or  unit  instructor  based  instruction  (IB);  part  task, 
computer  based  instruction  (CBI);  and  full  function  stand 
alone  trainers  (FFSAT) .  These  are  major  training 
alternatives  in  use  today  and  are  defined  in  the  next  four 
paragraphs . 

Classroom  or  unit  instructor  based  instruction  (A  ) 

1 

is  the  traditional  training  given  by  an  instructor.  Both 
small  group  instruction  (anywhere  from  1:6  to  1:16 
instructor  to  student  ratio)  and  large  group  instruction 
(1:17  to  ]:60  instructor  to  student  ratio)  are  included  in 
this  category.  Instructors  provide  instruction  through 
traditional  audio-visual  means.  This  includes 
blackboards,  overhead  projectors,  35mm  slide  projectors, 
films,  video  tapes,  guest  speakers  and  panel  discussions. 
Student  performance  evaluation  ranges  from  instructor 
subjective  opinion  of  classroom  contribution  through 
practical  exercises  to  graded  individual  papers  and 
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written  or  hands  on  (hands  on  the  actual  piece  of 
equipment)  performance  examinations. 

Part  task,  computer  based  instruction  (A  )  ranges 

2 

from  instruction  provided  by  a  table  top  personal  computer 
through  a  learning  station  linked  to  a  mainframe  computer 
to  a  table  top  computer  based  delivery  system  specifically 
designed  for  training.  This  latter  subcategory  includes 
systems  such  as  the  Electronic  Information  Delivery  System 
(which  provides  enhanced  learning  through  a  touch  screen 
display  vice  a  pure  keyboard  operation)  and  the  Video  Disc 
Gunnery  Simulator  (a  table  top  videodisc  based  trainer 
designed  to  familiarize  a  tank  gunner  with  the  fire 
control  equipment  at  his  ere w  station  and  train  the  tasks 
engage  the  target  with  the  main  gun  and  coax  machinegun 
from  the  gunner’s  station  under  varying  combat 
conditions).  Instructors  or  facilitators  are  present  for 
the  instruction  but  only  provide  answers  to  specific 
student  questions  on  lesson  content,  operation  of  the 
computer  providing  the  instruction,  or  direct  movement  to 
the  next  training  step  (when  the  computer  cannot  perform 
this  function).  This  category  can  Include  collective 
trainers  whose  function  is  to  train  partial  task  groupings 
or  parts  of  individual  tasks  without  fully  replicating  the 
actual  piece  of  equipment  or  crewmember  station.  A  good 
example  of  this  is  the  Combined  Arms  Tactical  Trainer 
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(often  referred  to  as  Simulation  Networking  -  SIMNET) . 

This  trainer  trains  crews  in  collective  gunnery  and 
tactical  tasks  by  providing  only  the  required  fidelity 
necessary  to  train  the  tasks  chosen  for  training. 

Full  function  stand  alone  trainers  (A  )  are 

3 

designed  to  replicate,  to  the  extent  technology  will 
allow,  performance  on  a  task  in  the  same  way  as  it  i3 
performed  on  the  operational  equipment.  They  are  also 
designed  to  replicate  the  physical  environment  of  the 
crewmember’s  station  (knobs,  controls,  switches,  seats, 
legroom,  etc.).  Good  examples  of  these  type  of  trainers 
are  flight  simulators  for  helicopters  and  fixed  wing 
aircraft  and  for  the  tank  or  Bradley  Fighting  Vehicle 
(BFV)  -  the  Unit  Conduct  of  Fire  Trainer  (UCOFT) .  The 
UCOFT  replicates  the  physical  characteristics  of  the 
vehicle  commander  and  gunner  positions  and  uses  a  computer 
driven  matrix  of  685  increasingly  difficult  exercises  to 
train  vehicle  commander  and  gunner  teams  on  target 
engagement  techniques.  These  stand  alone  trainers  are 
frequently  quite  similar  to  their  embedded  training 
counterparts  with  one  notable  exception.  It  is  usually 
not  possible  to  replicate  the  conditions  of  weather  which 
would  affect  human  performance  on  a  task.  For  example, 
flight  simulators  and  the  UCOFT  are  enclosed  in 
environmentally  controlled  rooms  or  shelters,  so  the 
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temperature  and  humidity  which  may  effect  a  tank  crew’s 


performance  on  a  task  outdoors  is  not  replicated  in  the 
simulator.  These  full  function  stand  alone  trainers  often 
have  limited  accessibility  since  they  would  not  be 
produced  in  the  same  quantities  as  would  the  operational 
equipment  (a  function  of  cost) .  Instructors  or  operators 
are  present  for  the  training  but  facilitate  the  students 
progress  through  the  training  software  rather  than  provide 
the  training  themselves.  These  trainers  would  either 
provide  their  own  performance  evaluation  and  feedback  or 
would  be  supplemented  by  an  instructor  or  operator  to 
provide  any  additional  required  evaluation  or  feedback  on 
task  performance. 

Embedded  training  is  the  fourth  alternative. 

Embedded  training  was  defined  in  Chapter  I.  Instructors 
or  facilitators  can  be  present  for  the  task  training 
provided  by  an  embedded  training  capability  but  do  not 
present  the  training.  They  perform  the  same  role  as  in 
alternatives  two  and  three.  Feedback  may  be  provided  by 
the  embedded  training  capability  itself  or  by  an 
instructor  or  facilitator  designated  to  observe  the 
training . 

The  method  goal  and  training  effectiveness  criteria 
were  established  in  Chapter  IV.  Now  that  we  have 
identified  the  training  alternatives  to  rank  we  can 
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complete  the  hierarchial  structure  (step  4) : 


FIRST  !  TRAINING  EFFECTIVENESS 

LEVEL  !OF  A  PROPOSED  TRAINING  SUBSYSTEM!  OBJECTIVE 


! STIMULI  !  ! FEEDBACK  !  ! PHYSICAL  ! 

.[PRESENTATION!  !  PRES  ENT  ATI  ON!  iilMILARITY! 

SECOND 

LEVEL  _  _  _ CRITERIA 

! RESPONSE  !  ! TASK  !  ! TRAINING! 

! PRESENTATION!  I  COMMONALITY !  ! TIME  ! 


[CLASSROOM  OR  UNIT!  ! PART  TASK  ! 

! INSTRUCTOR  BASED  !  ! COMPUTER  BASED! 

! INSTRUCTION  _!  ! INSTRUCTION  i 

THIRD 

LEVEL  _  _  ALTERNATIVES 

! FULL  FUNCTION!  ! EMBEDDiD ! 

! STAND  ALONE  !  1 TRAINING! 

UBAINgRS _ ! 

Step  5  of  the  training  effectiveness  comparison  method  is 
to  develop  the  questionnaire  and  obtain  the  subject  matter 
expert  pairwise  comparisons  used  to  establish  the 
effectiveness  of  the  four  training  alternatives  under  each 
of  the  six  criteria  for  each  of  the  eight  tasks  chosen.  I 
have  addressed  specifically  the  comparison  matrix  used  to 
establish  the  relationship  between  the  criteria  and  the 
goal  and  the  rating  scale  used  to  make  those  comparisons 


in  Chapter  IV.  The  general  alternative  versus  criteria 
matrix  shown  in  Chapter  IV  now  becomes: 


Criterion  1C1...C6) 

A 1 

A2 

A3 

A4  : 

A1 

1 

1 

1 

A2 

1 

1 

1 

A3 

1 

1 

1 

A4 

i  : 

So  we  have  six  comparison  matrices  (i.e.,  stimuli 
presentation,  response  presentation,  feedback 
presentation,  task  commonality,  physical  similarity,  and 
training  time)  which  require  six  comparisons  (A1:A2, 

A1:A3,  A1:A4,  A2 : A3 ,  A2 : A4 ,  A3:A4).  The  remaining  six 
comparisons  are  the  reciprocal  value  of  those  just 
established . 

Thus,  the  subject  matter  expert  must  make  15 
comparisons  to  establish  the  importance  of  the  six 
criteria  to  the  overall  goal  and  36  comparisons  for  each 
of  the  eight  tasks  considered.  The  questionnaire  used  is 
found  at  Appendix  B. 

The  questionnaire  return  rate  was  60%.  This  was 
lower  than  the  80%  anticipated.  Twelve  questionnaires  were 
mailed  for  data  collection.  Ten  returns  were  requested 
(the  two  extras  were  provided  in  the  off  chance  that 
additional  SMEs  might  be  available).  Of  these  ten,  six 
questionnaires  were  returned. 

Appendix  D  provides  the  subject  matter  experts’ 
responses  to  the  questionnaire  (Appendix  B) .  It  also 
provides  each  eigenvector  calculation  of  the  alternatives 
with  respect  to  each  of  the  six  criteria.  As  discussed  in 
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Chapter  IV,  the  eigenvector  method  solves  for  the  largest 
eigenvalue  to  eventually  obtain  the  weights  provided  in 
Appendix  D.  The  largest  eigenvalue  is  used  to  establish  a 
consistency  index.  The  consistency  index  provides  an 
assessment  of  the  degree  of  logic  in  the  subject  matter 
experts  Judgment  applied  in  completing  a  pairwise 
comparison  matrix.  The  training  effectiveness  comparison 
method  using  six  criteria  requires  six  pairwise  comparison 
matrices  for  each  of  the  eight  tasks  considered.  An 
additional  pairwise  comparison  matrix  is  required  for  each 
subject  matter  expert  to  establish  the  relative  importance 
(weights)  of  the  six  criteria  to  the  goal  (training 
effectiveness).  In  our  case  there  were  294  pairwise 
comparison  matrices  each  with  a  consistency  index.  For 
example  subject  matter  expert  #  1  evaluated  Task  A  by 
establishing  an  alternative  versus  criteria  comparison 
matrix  for  each  of  the  six  criteria.  Since  SME  #1 
evaluated  eight  tasks  his  total  number  of  alternative 
versus  criteria  comparison  matrices  was  (8) (6)  or  48 
comparison  matrices.  To  this  total  SME  also  generated 
a  criteria  versus  training  effectiveness  comparison  matrix 
to  establish  the  weights  of  the  criteria.  Therefore,  SME 
*1  established  48+1  or  49  total  comparison  matrices.  Data 
from  six  subject  matter  experts  were  used  in  this 
demonstration.  Therefore  the  total  number  of  comparison 
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matrices  was  (6) (49)  or  294.  Appendix  G  provides  a 
detailed  analysis  of  the  294  consistency  indexes.  Our 
interest  is  to  determine  the  extent  to  which  and  in  what 
proportion  the  consistency  indexes  exceed  .09  for  the  288 
alternative  versus  criterion  matrices  and  . 124  for  the  six 
criteria  versus  goal  matrices.  The  results  are  as 
follows:  (Note:  the  consistency  indexes  for  the  six 
criteria  versus  goal  matrices  are  included  in  this  summary 
and  are  also  shown  in  the  summary  of  the  results  of  the 
subject  matter  expert  comparisons  of  the  criteria): 
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then  we  have  captured  82.6%  (238/288)  of  the  data.  And  if 
we  expand  up  to  .3  then  89.6%  (258/288)  of  the  pairwise 
comparison  matrices  have  an  acceptable  consistency  index. 
The  six  consistency  indexes  from  the  criteria  versus  goal 
comparison  matrices  (see  first  chart  in  Chapter  VII)  were: 
.031,  .085,  .198,  .116,  .327,  and  .026.  Sixty-six  percent 

(4/6)  were  under  the  .124  consistency  index  level 
recommended  for  a  matrix  of  rank  6.  I f  we  expand  the 
range  to  .3  then  83.3%  (5/6)  of  the  pairwise  comparison 
matrices  have  an  acceptable  consistency  index. 
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CHAPTER  VII.  INDIVIDUAL  (TOPSIS)  AND  AGGREGATE  (BORDA) 
ALTERNATIVE  RANKINGS  (STEPS  6  &  7)  . 

Step  6  of  the  training  effectiveness  comparison 
method  is  to  establish  the  subject  matter  expert 
individual  training  alternative  rankings  using  the 
Technique  for  Order  Preference  by  Similarity  to  the  Ideal 
Solution  (TOPSIS) .  Appendix  E  provides  the  TOPSIS 
rankings  for  each  subject  matter  expert’s  evaluation  of 
each  of  the  eight  tasks.  Step  7  of  the  training 
effectiveness  comparison  method  aggregates  the  individual 
preference  rankings  into  an  overall  ranking  of 
alternatives  for  each  task  under  consideration  using  the 
Borda  aggregation  technique.  The  overall  ranking 
established  by  the  Borda  aggregation  of  the  six  subject 
matter  experts’  individual  rankings  is  found  at  Appendix 
F.  The  alternative  ranking  for  each  of  the  eight  tasks 
under  consideration  is  provided. 

The  results  will  be  summarized  shortly.  But  first  a 
discussion  of  the  criteria  weights  established  by  the 
subject  matter  experts  is  appropriate.  These  weights  must 
be  established  to  conduct  the  second  step  of  TOPSIS  - 
construct  a  weighted  normalized  decision  matrix.  The 
chart  below  summarizes  the  results  of  the  eigenvector 
estimation  of  the  subject  matter  experts’  judgment  on  the 
importance  of  each  criteria  with  respect  to  the  overall 


70 


46 


goal  of  achieving  an  effective  training  alternative: 
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FP=  Feedback  Presentation  SP=  Stimulus  Presentation 

PS=  Physical  Similarity  TC=  Task  Commonality 

RP*  Response  Presentation  TT*  Training  Time 

c=  consistency  index 

The  columns  of  this  chart  provide  each  subject  matter 
expert’s  weighted  criteria.  Rows  1  through  6  represent 
the  rank  order  of  each  criteria.  At  the  bcttom  of  each 
column  is  the  consistency  index  of  the  subject  matter 
expert’s  Judgement  for  the  ranking  and  weights 
established.  The  number  in  parenthesis  is  the  weight  for 
that  particular  criteria.  For  example,  SME  #1  ranked  task 
commonality  the  most  important  criteria  with  a  .39 
weight.  Training  time  was  ranked  least  important  with  a 


.04  weight. 
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An  example  will  illustrate  how  these  weights  were 
established.  The  answers  to  questions  1  through  15  of 
Section  III  of  the  questionnaire  (Appendix  B)  provide  the 
subject  matter  experts’  pairwise  comparisons  of  the 
criteria  with  regard  to  the  overall  goal  of  training 
effectiveness.  The  eigenvector  pairwise  comparison 
section  of  the  TOPSIS  output  sheet  in  Appendix  E  provides 
the  responses  of  each  of  the  six  subject  matter  experts  to 
those  15  pairwise  comparisons.  For  example,  subject 
matter  expert  *1  had  the  following  criteria  versus 
effectiveness  comparison  matrix: 

The  Eigenvector  Pairwise  Comparisons 


Sti 

Res 

Foe 

Tas 

Phy 

Tng 

Stimuli 

1  0 

1 . 0 

1 .0 

.  33 

3.0 

4 . 0 

Response 

1 . 0 

1 . 0 

1 .0 

.33 

3 . 0 

4 . 0 

Feedbac 

1 . 0 

1 .0 

1 . 0 

.  33 

3 . 0 

4 . 0 

Task  Co 

3 . 0 

3 . 0 

3 . 0 

1 . 0 

5.0 

5 . 0 

Phy  Sim 

.  33 

.  33 

.  33 

.  20 

1 . 0 

3 . 0 

Tng  Tim 

.  25 

.  25 

.  25 

.  20 

.  33 

1 . 0 

The  DM  subjective 

weights 
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o  f 

the  SME  * 

1  TOPSIS 

output  sh 
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ide  the 

resul t i 

ng  e 

i genvector 

estimat 

of  those 

weights : 

The  DM  subjective 

weights 

S  t i mu 1 i 

Respons 

Feedbac  Task 

Co 

Phy  Sim 

Tng  Tim 

0  .  16 

0 . 16 

C  .  16 

0  . 

39 

0 . 07 

0 . 04 

These  weights  are  the  numbers  in  parenthesis  shown  in  the 
subject  matter  expert  weights  summary  chart  for  SME  #  1. 
The  TOPSIS  individual  ranking  results  provided  in 
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Appendix  E  and  the  Borda  aggregation  of  those  rankings 
found  in  Appendix  F  are  summarized  for  each  of  the  eight 
tasks . 


TASK  A:  PERFORM  BEFORE  OPERATIONS  CHECKS  AND  SERVICES 


R 

A 

N 

K 

I 

N 

G 


SME  INDIVIDUAL  RANKING 
TOPSIS 


#  1 

#2 

#3 

#4 

IB ( .61) 

ET ( . 67) 

ET  (  1 . 0) 

IB( .71) 

ET ( .49) 

SA ( . 54) 

SA ( . 40) 

CB ( . 44) 

S A ( . 47) 

IB ( . 36) 

CB  (  .  1 1) 

SAC . 43) 

CB ( . 12) 

CB ( . 14) 

IB ( . 00) 

ET ( . 32) 

SME 

SME  AGGREGATE 

1 

2 

3 

4 


INDIVIDUAL  RANKING 


RANKING 


R 

A 

topsis  : 

*5  :  #6  : 

Borda 

N 

; 

K 

1 

ET ( . 58) 

ETC. 84)  ! 

ET  (14) 

I 

2 

CB ( . 56) 

SAC. 77)  : 

SA  (  9) 

N 

3 

SAC . 35) 

CB (.19)  : 

IB  (  7) 

G 

4 

IB  C . 03) 

IBC.16)  i 

CB  (  6) 

CB=  Computer  Based  Instruction  ET=  Embedded  Training 
IB=  Instructor  Based  Instruction 
SA=  Stand  Alone  Trainer 


This  chart  provides,  for  Task  A,  the  individual  SME  (  #1- 
#6)  training  alternative  rankings  in  each  column.  The 
last  column  provides  the  Borda  aggregate  ranking  for  all 
six  subject  matter  experts.  The  rows  are  rank  ordered  1 
through  4.  The  number  in  parenthesis  is  the  TOPSIS  c 
value  or  relative  closeness  to  the  ideal  solution.  For 
example,  SME  *1  has  instructor  based  training  ranked  first 
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with  a  c  value  of  .61.  His  last  choice  is  computer  based 


instruction  with  a  c  value  of  .12.  The  number  in  the 
parenthesis  of  the  Borda  aggregate  column  is  the  Borda 
score  awarded  to  that  alternative.  In  this  case,  ET  was 
selected  first  with  a  Borda  score  of  14. 

Again,  an  example  will  better  illustrate  how  these 
results  were  established.  Let  us  look  at  SME  #  1  and  his 
evaluation  of  the  training  alternatives  with  respect  to 
Task  A.  The  answers  to  questions  1  through  36  under  Task 
A  of  Section  V  of  the  questionnaire  (Appendix  B)  provide 
the  subject  matter  expert’s  pairwise  comparisons  of  the 
four  training  alternatives  with  regard  to  each  of  the  six 
criteria  for  Task  A.  These  six  comparison  matrices  for 
SME  *  l’s  evaluation  of  Task  A  are  provided  by  the  first 
data  sheet  of  Appendix  D.  Note  that  the  responses  to 
questions  1  through  36  are  provided  by  the  top  diagonal 
halves  of  the  six  matrices.  The  weights  of  each 
alternative  with  respect  to  each  criteria  are  established 
using  the  eigenvector  estimation  method.  The  DM 
subjective  weight  section  provides  those  weights  (e.g.  the 
weights  established  for  the  alternatives  versus  stimuli 
presentation  are:  IB  -  .05,  CBI  -  .15,  FFSAT  -  .35,  ET  - 
.46) .  These  values  are  used  to  establish  the  decision 
matrix  required  by  Step  1  of  TOPSIS.  The  first  TOPSIS 
output  sheet  (Appendix  E)  demonstrates  how  this  decision 
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matrix  is  formed.  The  decision  matrix  is  found  in  the 
first  section  of  the  output  sheet.  Notice  that  the  first 
column  of  the  matrix  -  stimuli  presentation  -  has  the  same 
values  as  the  weights  established  by  the  eigenvector 
estimation  for  the  alternatives  versus  stimuli 
presentation  shown  above.  The  remaining  5  columns  of  the 
matrix  are  completed  using  the  DM  subjective  weights  for 
the  second  through  sixth  comparison  matrices  found  on  the 
first  output  sheet  in  Appendix  D. 

We  can  follow  the  progression  of  Steps  3  through  6  of 
TOPSIS  by  returning  to  the  first  TOPSIS  output  sheet  in 
Appendix  E  .  Step  3  is  provided  by  the  ideal  solution  and 
negat ive- ideal  solution  section  of  the  output  sheet. 

Steps  4  and  5  are  calculated  within  the  program.  Step  6  - 
rank  the  preference  order  -  is  provided  by  the  relative 
closeness  to  the  ideal  solution  section  of  the  output 
sheet.  In  our  example  the  ranking  was  established  by 
decreasing  relative  closeness  to  the  ideal  solution  or  c 
value  as  shown:  instructor  based  instruction  -  .61, 

embedded  training  -  .49,  stand  alone  trainer  -  -47,  and 
computer  based  instruction  -  .12.  This  is  the  same 
ranking  and  c  value  listing  shown  for  SME  #  1  under  the 
subject  matter  expert  summary  chart  for  Task  A  The 
final  column  of  the  summary  chart  provides  the  Borda 
ranking  (first  section)  found  in  Appendix  F.  Note  that 
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the  Borda  points  shown  in  parenthesis  of  the  summary  chart 
match  those  shown  in  the  Task  A  section  of  Appendix  F. 


TASK  B:  TROUBLESHOOT  THE  FIRE 
CONTROL  SYSTEM 


SME  INDIVIDUAL  RANKING 
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#4 
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IB  (  .  61) 

ETC .71) 

ET ( 1 . 0) 
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ETC .49) 
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CB ( . 67) 
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CB ( . 12) 
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IB ( . 00) 
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CB ( .68) 
ETC .57) 
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CB ( . 18) 
IBC .03) 

ET  (14) 

SA  (  8) 

IB  (  7) 

CB  (  7) 

CB=  Computer  Based  Instruction  ET=  Embedded  Training 
IB=  Instructor  Based  Instruction 
SA=  Stand  Alone  Trainer 

TASK  C:  EVACUATE  A  WOUNDED  CREWMAN 
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TASK  D:  REPLACE  A  THROWN  TRACK 
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TASK  E:  ENGAGE  TARGETS  WITH  THE  MAIN  GUN  FROM 
THE  GUNNER'S  STATION 
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SME  SME  AGGREGATE 

INDIVIDUAL  RANKING  RANKING 
R  ! :  TOPSIS  ! !  Borda 
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CB=  Computer  Based  Instruction  ET=  Embedded  Training 
IB=  Instructor  Based  Instruction 
SA=  Stand  Alone  Trainer 

TASK  F:  ADJUST  CREW  COMPARTMENT  PRESSURIZATION 
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TASK  G:  ZERO  THE  MACHINEGUN 
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CB=  Computer  Based  Instruction  ET=  Embedded  Training 
IB=  Instructor  Based  Instruction 
SA=  Stand  Alone  Trainer 

TASK  H:  ENGAGE  TARGETS  WITH  THE 
COAX  FROM  THE  GUNNER’S  STATION 
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CB=  Computer  Based  Instruction  ET=  Embedded  Training 
IB=  Instructor  Based  Instruction 
SA=  Stand  Alone  Trainer 


TOPSIS  alternative  rankings  are  based  on  the  relative 
closeness  (c)  to  the  ideal  solution  which  is  a  measure  of 
both  the  alternatives’  closeness  to  the  ideal  solution  and 


separation  from  the  negative-ideal  solution.  When  c=l 
then  that  particular  alternative  is  the  ideal  solution 
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« 

(A  =A  )  and  when  c=0  that  alternative  is  the 
i 

negative- ideal  solution  (A  =A  ) .  Task  A  provides  an 

i 

example  of  this.  For  SME  *3,  embedded  training  has  a  c 
value  of  1  and  instructor  based  instruction  has  a  c  value 
of  0.  In  this  case  embedded  training  is  the  ideal 
solution  and  instructor  based  training  is  the 
negative-ideal  solution.  The  c  value  also  gives  an 
indication  of  how  much  better  one  alternative  is  over 
another  and  how  close  it  is  to  the  best  (ideal)  solution. 
However,  this  'betterness'  is  related  to  our  rating  scale 
rather  than  an  absolute  measure  of  training 
effectiveness.  Our  rating  scale  ranges  from  equal 
importance  to  absolute  importance  instead  cf  time  in 
seconds  or  absolute  degree  of  task  physical  similarity 
with  an  alternative.  One  cannot  predict  the  true  outcome 
in  terms  of  achieved  training  effectiveness  when  an 
operator  uses  a  particular  alternative  to  train  on  a 
task.  In  task  C,  SME  #1  has  ranked  embedded  training 
number  1  (.89),  stand  alone  trainer  number  2  (.81)  and 

instructor  based  instruction  number  3  (.47).  This  tells 
the  decisionmaker  that  embedded  training  has  a  slight 
advantage  (.08)  over  a  stand  alone  trainer  and  a 
significant  advantage  (.42)  over  instructor  based 
training . 

The  final  column  is  the  aggregate  measure  of  the 
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individual  rankings  for  each  task.  This  aggregate  ranking 
is  provided  by  the  Borda  function.  The  number  in 
parenthesis  provides  the  sum  of  the  points  awarded  for 
1st,  2nd,  3rd,  and  4th  place  finishes.  This  point  total 
also  gives  the  decisionmaker  an  idea  of  the  'goodness'  of 
an  alternative.  This  is  provided  both  by  the  point 
separation  of  the  alternatives  and  the  number  12.  Twelve 
is  the  maximum  value  for  a  first  place  finish  with  four 
SMEs  and  four  alternatives  [4(m-l)=  4(3-1)=  12].  For 
example,  embedded  training  is  more  strongly  preferred  by 
the  subject  matter  experts  evaluation  of  Task  B  (9)  than 
it  is  for  Task  A  (8)  or  Task  C  (8) .  Embedded  training  for 
Task  B  is  also  more  preferred  against  instructor  based 
instruction,  stand  alone  trainers,  and  computer  based 
instruction  than  it  is  for  the  same  comparisons  of  point 
totals  for  tasks  A  and  C.  Again,  we  still  cannot  predict 
the  training  effectiveness  which  will  be  achieved  by  the 
operator.  However,  since  the  individual  rankings  are 
based  against  criteria  which  are  components  of  an 
effective  training  alternative  the  method  does  give  a 
decisionmaker  a  comparison  ranking  based  upon  training 
ef  f ectiveness . 
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CHAPTER  VIII..  CONCLUSIONS. 

This  chapter  draws  conclusions  from  the  results  of 
the  method  demonstration  provided  in  Chapter  VII.  In 
addition  to  the  overall  conclusion  the  discussion  includes 
consistency,  criteria  weights,  use  of  TOPSIS  and  Borda, 
questionnaire  sample  size,  combination  of  training 
effectiveness  and  cost,  use  of  the  method  for  project 
updates,  and  flexibility  of  the  method. 

The  major  conclusion  from  the  results  of  the 
demonstration  of  this  method  is  that  it  provides  a 
workable  solution  to  the  problem  of  investigating  training 
alternatives,  especially  embedded  training,  early  in  a 
system’s  development.  It  is  not  a  simple  method.  But  it 
does  provide  a  framework  for  the  analyst  to  organize  the 
systematic  study  of  the  training  alternative  question. 

The  recommendations  provided  in  the  next  chapter  provide 
an  opportunity  to  reduce  the  amount  of  time  necessary  to 
apply  this  method  to  a  developing  system. 

The  results  of  the  consistency  index  analysis  give  a 
guarded  approval  to  the  application  of  the  training 
effectiveness  comparison  method.  It  is  guarded  because 
38.1%  (112/294)  were  beyond  the  range  recommended  by  Saaty 
(.09  and  .124).  This  means  that  conflicts  in  just  over 
one-third  of  the  data  exist  to  force  inconsistency.  The 
question  now  becomes  how  much  over  the  Saaty  consistency 
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Index  levels  are  we  willing  to  accept.  In  Chapter  4  the 
random  indexes  for  matrices  of  rank  four  and  six  were 
discussed.  These  Indexes  (.9  and  1.24)  were  the  average 
consistency  indexes  of  500  randomly  generated  pairwise 
comparison  matrices.  Saaty’s  recommended  consistency 
index  of  .09  for  a  matrix  of  rank  4  (alternative  versus 
criterion  matrix)  represents  the  top  11%  of  the  most 
consistent  matrices  in  the  random  sample.  If  we  were 
willing  to  settle  for  less  consistency,  say  the  top  20  or 
30  percent  of  consistency  indexes  in  an  estimated 
population  of  4  X  4  matrices  scaled  1  to  9  (which  is  our 
case)  then  the  consistency  index  level  drops  to  . IP  and 
.27.  The  .27  level  includes  all  but  10.7%  (31/288;  of  the 
alternative  versus  criterion  matrices.  Since  the  30 
percent  level  includes  a  high  percentage  of  the  best 
consistency  indexes  within  the  estimated  population  of 
possible  4X4  matrices  we  can  conclude  that  our  data  are 
sufficiently  close  to  Saaty's  recommended  consistency 
level  to  be  useful. 

Therefore,  the  consistency  indexes  do  suggest  that 
the  pairwise  comparisons  developed  in  the  training 
effectiveness  comparison  method  can  produce  usable 
results.  Ninetv  percent  of  the  consistency  indexes  are 
within  a  reasonable  level  (top  one-third)  of  consistency 
and  Just  over  60%  of  the  consistency  indexes  were  below 
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Saaty  ’  s  recommended  level  of  ‘good’  consistency. 

Yet  sources  of  consistency  errors  did  exist.  First, 
subject  matter  experts  may  not  have  completely  understood 
the  definitions  of  the  training  alternatives,  the 
criteria,  or  the  tasks  and  how  they  affect  the  new 
system.  This  may  have  caused  subtle  changes  in  pairwise 
comparisons  from  one  alternative  (or  one  criteria)  to  the 
next.  Second,  some  subject  matter  experts  may  have  tried 
to  generalize  how  the  alternative  or  criteria  was  affected 
by  initial  or  sustainment  training  rather  than  focusing  on 
one  or  the  other  (Note:  embedded  training  is  normally 
associated  with  sustainment  training) .  Since  initial  and 
sustainment  training  can  affect  the  alternative  versus 
criterion  relationship  differently,  this  may  have  also 
caused  Judgment  inconsistencies  in  the  pairwise  comparison 
matrices.  Third,  the  criteria  are  not  completely 
independent.  It  is  possible  therefore  that  in  some 
Judgments  the  evaluation  of  an  alternative  with  respect  to 
each  criteria  may  overlap.  The  extent  to  which  this 
occurs  is  dependent  upon  the  subject  matter  expert’s 
understanding  of  the  criteria.  Finally,  from  looking  at 
the  repetitive  nature  of  the  responses  toward  the  end  of 
the  long  questionnaire  it  is  apparent  that  the  motivation 
of  the  subject  matter  expert  decreased  over  time.  This 
probably  was  normally  caused  by  competing  work  priorities 
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rather  than  an  unwillingness  to  complete  the  questionnaire 
properly . 

Two  conclusions  may  be  reached  when  interpreting  the 
results  of  the  criteria  weights  established  by  the 
subject  matter  experts.  First,  no  one  criterion  clearly 
dominates  another  (in  fact  many  are  equally  weighted) . 
Second,  SME  #3  and  SME  *5  have  high  consistency  index 
values .  If  time  had  been  available  these  subject  matter 
experts  should  have  remade  their  criteria  pairwise 
comparisons . 

The  questionnaire  sample  size  is  a  tradeoff  between 

obtaining  as  much  subject  matter  expertise  (i.e.  numbers 

of  SMEs)  and  the  practical  reality  of  who  is  accessible, 

able,  and  willing.  At  a  service  school  with  a  mission  of 

developing  a  new  system  there  are  probably  some  30 

personnel  who  fit  the  qualifications  for  an  SME  capable  of 

making  training  effectiveness  comparisons.  Those 

qualifications  are  expertise  in  a  particular  technical 

area,  training  alternatives,  and  the  new  system  under 

investigation.  It  is  possible  to  invoke  statistical 

sampling  rules  to  precisely  determine  what  sample  size  is 

large  enough  to  predict  the  effectiveness  outcome  within  a 

certain  margin  of  error  and  a  probability  of  exceeding 
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that  margin  of  error  for  the  entire  population. 
Unfortunately,  in  our  application,  this  would  be  a  tedious 
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process  of  determining  the  variances  of  questionnaire 
responses.  This  is  not  a  necessary  step  since  at  this 
early  stage  of  the  development  cycle  we  are  not  trying  to 
predict  actual  effectiveness  in  the  hands  of  the 
operator.  What  we  are  trying  to  do  is  provide  the 
decisionmaker  with  enough  data  to  make  an  informed 
decision.  Given  the  availability  of  subject  matter 
experts  at  the  various  service  schools,  ten  to  fifteen 
subject  matter  experts  will  provide  a  conservative  sample 
size  which  represents  one-third  to  one-half  of  the  total 
population  and  provides  an  adequate  sample  size  for  a 
major  study. 

TOPSIS  is  one  of  a  class  of  mul  t  i attr  ibute  decision 

making  models  for  which  cardinal  information  on  the 

attributes  is  available.  Others  in  the  class  are  the 

Linear  Assignment  Method,  the  Simple  Additive  Weighting 

Method,  the  Hierarchial  Additive  Weighting  Method,  and  the 
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Elimination  and  Choice  Translating  Reality  (ELECTRE) . 
TOPSIS  is  an  effective  model  to  use  out  of  this  class 
because  it  is  simple  to  use  and  provide:  the  best  chance 
of  a  distinct  alternative  ranking.  The  Borda  function 
technique  is  one  of  several  available  methods  for 
aggregating  alternative  rankings.  It  was  selected  because 
the  technique  provides  credit  for  each  place  finish  except 
for  last  place.  Tt  accounts  for  the  positive  Judgement  in 
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favor  of  an  alternative  without  penalizing  an  alternative 
for  votes  against  it  (which  some  methods  specifically 
do) .  The  point  I  want  to  stress  here  is  that  both  methods 
are  appropriate  for  this  type  of  problem.  They  are  not 
the  only  method  which  can  be  used  to  provide  the 
individual  and  aggregate  rankings.  An  investigation  into 
the  question  of  which  method  is  the  best  for  this 
application  is  not  necessary  since  several  methods  will 
provide  equally  valid  results.  This  is  especially  true 
when  one  remembers  that  the  intent  of  this  method  is  to 
obtain  an  estimated  comparison  of  training  alternatives  at 
a  stage  when  much  concrete  descriptive  information  is  not 
aval lable . 

Armed  with  alternative  rankings,  the  next  step  is  to 
fold  these  rankings  into  early  estimates  of  the  cost  of 
the  various  training  alternatives  for  the  developing 
system.  This  is  arguably  difficult  to  do  since  not  only 
is  the  cost  data  an  estimate  but  the  ef f ecti veness 
comparisons  generated  by  this  method  do  not  transform  to 
an  actual  prediction  of  how  effective  one  alternative  will 
be  (the  method  predicts  that  one  alternative  will  be  more 
effective  than  another).  This  makes  it  difficult  to 
establish  a  dollar  relationship  to  training  effectiveness. 
For  example,  if  our  alternative  has  a  c  value  of  .8  and  is 
compared  to  an  alternative  which  has  a  c  value  of  .4,  we 
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have  a  general  feeling  that  the  first  alternative  is  twice 
as  effective.  Since  we  are  not  predicting  the 
alternatives  actual  effectiveness,  we  are  not  able  to  say 
that  we  are  only  willing  to  pay  half  as  much  for  the 
second  alternative. 

This  method  can  also  be  used  for  project  updates 
beyond  its  initial  use  for  the  first  decision.  Just  as 
Phase  II  of  the  ARI  screening  process  is  implemented  when 
more  information  becomes  available  so  can  the  judgments 
from  the  subject  matter  experts  be  updated  as  more 
information  become*  available.  When  actual  training 
alternative  hardware  is  produced  and  can  be  tested  with 
the  operational  system  then  this  method  is  no  longer  the 
preferred  method  to  obtain  an  effectiveness  comparison. 
Actual  performance  data  should  then  be  used  to  Judge  the 
effectiveness  of  the  alternative  under  study. 

Finally,  a  key  advantage  of  this  method  is  its 
flexibility.  The  hierarchial  structure  of  the  method 
permits  the  analyst  determine  both  the  criteria  and 
alternatives  to  use  for  this  early  comparison  of  the 
effectiveness  of  training  subsystems.  I  have,  based  upon 
research  into  current  methods  of  predicting  training 
effectiveness,  presented  the  criteria  I  believe  are  most 
appropriate  for  this  comparison  problem.  But  changing  the 
criteria  with  adequate  Justification  does  not  destroy  the 
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solution  methodology.  More  importantly,  the  four 
alternatives  used  in  the  method  demonstration  can  be 
changed  to  include  the  training  alternatives  appropriate 
for  the  developing  system  under  investigation. 
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CHAPTER  IX.  RECOMMENDATIONS  AND  FUTURE  RESEARCH. 

The  training  effectiveness  comparison  model  goes 
beyond  current  methods  of  assessing  the  effectiveness  of 
training  alternatives  early  in  the  development  of  a  new 
system.  It  therefore  provides  a  useful  tool  during  the 
conduct  of  a  preliminary  training  effectiveness  analysis 
and  should  be  implemented.  Its  actual  utility  will  be  in 
large  part  determined  by  the  time  available  to  complete 
the  study  and  the  availability  of  subject  matter  experts 
to  participate  in  the  data  collection  process.  For  a 
system  with  a  large  number  of  tasks  the  method  requires  a 
significant  amount  of  time  to  collect  and  analyze  the 
data.  While  this  may  be  a  disadvantage  to  analysts  with 
limited  time  it  is  also  an  advantage  because  it  forces  an 
orderly  detailed  investigation  into  the  training  subsystem 
of  a  new  system  at  precisely  the  time  when  the 
consideration  of  embedded  training  requires  it  -  early. 

In  addition  to  the  general  recommendation  that  the 
method  be  implemented  the'-3  ire  several  specific 
recommendations  that  can  be  made  as  a  result  of  the 
demonstration  of  the  method.  These  are  in  the  areas  of 
data  collection,  the  separation  of  initial  and  sustainment 
training,  and  the  form  of  the  questionnaire. 

Several  of  the  subject  matter  experts  indicated  the 
questionnaire  was  too  long.  One  believed  his  consistency 
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suffered  because  he  was  forced  to  complete  the 
questionnaire  in  several  increments  without  refocusing  on 
the  criteria  or  tasks  under  cons iderat i on .  Additionally, 
the  repetitive  nature  of  some  of  the  data  in  the  last 
several  tasks  evaluated  indicates  that  the  respondents 
tend  to  lose  their  focus  on  the  pairwise  comparison 
process  after  several  hours  of  answering  questions.  For 
these  reasons,  I  believe  the  best  way  to  collect 
comparison  data  of  this  nature  is  in  short  spans  of  time. 
Therefore  I  recommend  a  shortened  three  part  collection 
scheme.  First,  tasks  should  be  evaluated  in  groups  of 
three  to  four  and  if  possible  cover  the  same  area  of  the 
operational  system.  Second,  combat  developers  in 
coordination  with  the  contractor  for  the  system  should 
provide  a  briefing  on  everything  known  about  that  portion 
of  the  system  that  affects  the  selected  group  of  tasks. 
This  briefing  would  be  presented  to  the  subject  matter 
experts  (these  are  experts  on  training  vice  the  system) 
chosen  for  the  preliminary  training  effectiveness  study 
and  last  thirty  to  sixty  minutes.  Finally,  about  one  to 
two  hours  would  then  be  spent  by  the  subject  matter 
experts  to  establish  their  pairwise  comparisons  on  the 
selected  group  of  tasks.  The  idea  is  not  to  overwhelm  the 
experts  who  tend  to  lose  concentration  beyond  two  hours  of 
pairwise  comparisons.  This  short  spurt  also  has  the 
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possible  advantage  of  keeping  the  subject  matter  experts 
focused  on  the  specific  tasks  under  consideration.  The 
disadvantage  of  this  approach  is  that  it  extends  the 
number  of  days  required  to  complete  the  evaluation  of  a 
major  system.  For  example,  the  Future  Armored  Combat 
System  has  274  individual  and  collective  tasks.  Thl3 
means  a  complete  evaluation  of  the  system  would  take  at 
least  69  days  (or  14  weeks  &  5  days  a  week)  for  the  data 
collection  alone.  Therefore  a  study  of  this  magnitude 
should  be  scheduled  over  a  period  of  about  six  months. 

The  first  two  months  would  be  used  to  identify  training 
alternatives,  design  the  questionnaire,  and  coordinate  for 
data  collection  Data  collection  would  be  conducted  over 
the  next  three  months.  Data  reduction,  data  analysis  and 
preparation  of  the  final  report  would  occur  during  the 
final  two  months. 

The  second  recommendation  recognizes  the  difference 
in  the  types  of  learning  involved  in  initial  training  and 
sustainment  training.  The  method  presented  in  this 
project  can  apply  in  both  cases  although  the  distinction 
was  not  made  in  the  demonstrati  on  of  the  method.  Future 
applications  of  the  model  ought  to  make  this  distinction. 
The  training  effectiveness  criteria  are  still  applicable 
but  their  relationship  to  the  training  alternatives  often 
changes  when  one  considers  the  best  way  to  impart  initial 
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knowledge.  When  evaluation  time  is  short  the  focus  ought 
to  be  on  sustainment  training  since  the  unit  setting  is 
often  the  most  cost  effective  environment  for  embedded 
training.  This  is  so  because  of  the  large  number  (often 
one  per  student  in  a  class)  of  operational  systems  that 
would  be  required  to  support  an  embedded  training  approach 
in  a  school  system. 

Third,  the  questionnaire  as  currently  designed  is 
overly  repetitive.  One  respondent  suggested  that  the 
generic  question  be  posed  and  then  followed  by  a  chart 
version  of  the  required  six  comparisons.  For 
example : 

STIMULI  PRESENTATION: 

To  what  degree  does  the  first  training 
alternative  have  to  present  information  required  to  train 
the  _linsert_taskX_  when  compared  to  the  second 
alternative  [1-9]? 

_  Instr  based  instr  :  comp  based  instr  _ 

_  Instr  based  instr  :  stand  alone  trnr  _ 

_  Instr  based  instr  :  embedded  training 

_  Comp  based  instr  :  stand  alone  trnr  _ 

_  Comp  based  instr  :  embedded  training  _ 

_  Stand  alone  trnr  :  embedded  training  _ 

This  form  of  the  questionnaire  condenses  the  information 
presented  and  no  longer  uses  the  reciprocal  rating  scale. 
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Respondents  need  only  determine  which  alternative  is 
projected  as  dominant  then  place  their  rating  on  the  side 
of  the  dominant  rating  scale.  For  example  the  entry: 

_ 3__  Instr  based  instr  :  comp  based  instr _ - _ 

would  mean  that  instructor  based  instruction  is  dominant 
and  is  slightly  favored  over  computer  based  instruction. 
This  would  shorten  the  time  required  to  complete  the 
questionnaire  and  may  well  improve  the  consistency  of  the 
response  by  providing  the  subject  matter  expert  with  a 
compressed  presentation  of  his  responses. 

Future  study  should  focus  on  the  implications  of  the 
method’s  use  when  an  alternative  is  added  or  deleted  from 
the  Initial  set  of  alternatives,  on  the  importance  of  one 
level  of  consistency  over  another,  and  on  the  method’s 
ability  to  predict  actual  training  effectiveness.  It  is 
quite  possible  that  at  some  stage  of  the  study  or  the 
developmental  cycle  it  may  be  necessary  to  delete  an 
alternative  from  further  consideration  or  to  add  a  new 
alternative.  Alternatives  may  be  deleted  because  of  a 
failure  of  a  key  technology  to  mature  or  because  the  cost 
becomes  too  high.  Alternatives  may  be  added  because  a 
technology  matures  faster  than  expected  or  additional 
dollars  are  available  to  the  program.  When  an  alternative 
is  deleted  it  should  only  require  the  analyst  to  delete 
the  comparisons  made  with  the  deleted  alternative.  In  our 
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case  if  we  delete  full  function  stand  alone  trainers  (A3) 


from  one  of  the  tasks  under  cons iderat i on  the  original 
pairwise  comparison  matrix: 


Criterion  (C1...C6) 

A 1 

A2 

A3 

a  4  : 

A1 

1 

1 

1 

A2 

1 

I 

i 

A3 

1 

1 

1 

A4 

1  ! 

becomes : 

Criterion  (Cl  . 

.  .C6)  ! 

A1 

A2 

A4  : 

A1 

1 

1 

1 

1 

I 

A2 

1 

1 

1 

1 

1 

_ A4 _ 

• 

1 

1  1 

It  should  be  a  simple  matter  to  Just  delete  the  A1:A3, 
A2:A3,  and  A3 : A4  comparisons  from  the  data  set  and 
recalculate  the  eigenvector  weights  for  the  alternatives 
remaining  (for  ea^h  criterion).  The  question  further 
research  should  investigate  is  whether  or  not  the  deleted 
alternative  or  added  alternative  would  have  influenced  the 
rating  scale  used  by  the  subject  matter  expert. 

The  true  measure  of  the  importance  of  one  level  of 
consistency  over  another  for  a  matrix  of  a  particular  size 
(in  our  case  a4X4ora6X6)  may  be  the  point  at  which 
higher  consistency  index  values  change  the  rank  order 
outcome  provided  by  TOPSIS.  Additional  research  into  this 
question  for  our  specific  application  may  well  provide  the 
most  appropriate  consistency  level  beyond  which  pairwise 
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comparison  matrices  must  be  regenerated. 

Finally,  we  know  from  earlier  discussion  that  the 
method  does  not  predict  actual  training  performance  that 
will  result  from  the  operator  using  a  particular  training 
alternative.  If  a  correlation  between  TOPSIS  c  values  or 
Borda  points  and  the  actual  performance  achieved  can  be 
established,  then  the  predictive  validity  of  this  method 
can  be  determined  and  more  direct  cost  effectiveness 
relationships  can  be  established. 
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APPENDIX  A:  FUTURE  ARMOR  COMBAT  SYSTEM  TASK  LIST. 


DRIVER 

Start/Stop  the  Engine 
Drive 

Slave  Start 

Perform  Fuel  Transfer  Procedures 
Recover  Tank  by  Similar  Vehicle 
Prepare  Driver’s  Station  for  Operation 
Secure  Driver’s  Station 

Operate  the  Night  Vision  Viewer  in  Driver’s  Hatch 
Remove/Instal 1  Track  Block(s) 

Troubleshoot  the  Night  Vision  Viewer 
Troubleshoot  using  Driver’s  Control  Panel  Warning 
and  Caution  Lights 
Troubleshoot  the  Engine 
Troubleshoot  the  Transmission 
Troubleshoot  the  Personnel  Heater 
Troubleshoot  the  Gas  Particulate  Filter  Unit 
Perform  Before-Operation  Checks  and  Services 
Perform  During-Operation  Checks  and  Services 
Perform  After-Operation  Checks  and  Services 
Stow  Ammunition 
Establish  Silent  Watch 
Evacuate  a  Wounded  Crewman 
Extinguish  a  Fire 


A 
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Service  the  Precieaner 


Tow  a  Tank 

Replace  a  Thrown  Track 

Operate  the  Gas  Particulate  Filter  Unit 
Refuel  the  Tank 
Unlock  Stuck  Parking  Brakes 
Service  the  Air  Induction  System 
Prepare  for  Power-pack  Removal 

Perform  Eef ore-Operation  Checks  and  Services  Tank  NBC 
System 

Inspect  Main  Gun  Ammunition  for  Serviceability 

Stow  Main  Gun  Ammunition 

Clean/Service  the  Main  Gun 

Inspect  the  Hydraulics 

Troubleshoot  the  NBC  System 

Place  the  NBC  System  into  Operation 

Adjust  Crew  Compartment  Pressurization 

Stow  Ammunition 

Perform  Operator  Maintenance  Tank  Periscopes 
Hook-up/Operate  Mine  Plow/Roller  Assembly 
Prepare  Vehicle  for  fording  Operations 
Operate  Over-Pressurization  System 
Load/Unload  a  Grenade  Launcher 

Perform  Pre-operation  Checks  and  Services  (Auxiliary 
Power  Unit)  APU 
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Perform  Before-Operation  Checks  and  Services-APU 
Perform  During-Operation  Checks  and  Services-APU 
Perform  After-Operation  Checks  and  Services-APU 
Troubleshoot  the  APU 
Start  Generator  -  APU 

Shut  down  the  Generator  and  Engine  -  APU 
GUNNER 

Recover  Tank  by  Similar  Vehicle 
Remove/ Instal 1  Track  Block(s) 

Perform  Before-Operation  Checks  and  Services 
Perform  During-Operation  Checks  and  Services 
Perform  After-Operation  Checks  and  Services 
Perform  Gunner’s  Preventive  Maintenance  Prepare-to- 
fire  Checks  and  Services 
Perform  Gunner’s  Preventive  Maintenance  After- 
fire  Checks  and  Services 
Prepare  Gunner’s  Station  for  Operation 
Secure  Gunner’s  Station 
Troubleshoot  the  Fire  Control  System 
Prepare  Gunner’s  Weapon  for  Travel 
Evacuate  a  Wounded  Crewman 
Extinguish  a  Fire 
Replace  a  Thrown  Track 

Operate  the  Gas  Particulate  Filter  Unit 
Boresight  and  System  Calibrate 
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Perform  Manual  Extraction  of  a  Main  Gun  Round 
Prepare  for  Powerpack  Removal 

Perform  Operator  Before  Operations  Checks  and  Services 
NBC  System 

Inspect  Main  Gun  Ammunition  for  Serviceability 

Stow  Main  Gun  Ammunition 

Clean/Service  the  Main  Gun 

Inspect  the  Hydraulics 

Refuel 

Engage  Targets  with  the  Main  Gun  from  the  Gunner’s 
Station 

Perform  Operator  Maintenance  on  Periscopes 
Place  the  NBC  System  into  Operation 
Adjust  Crew  Compartment  Pressurization 
Stow  Ammunition 
Troubleshoot  the  NBC  System 
Troubleshoot  the  Main  Gun 

Adjust  the  Operating  Cam  on  the  Main  Gun 
Load/Unload  a  Grenade  Launcher 
Instal 1 /Remove  a  Coax  Machinegun 
Zero  the  Machinegun 

Engage  Targets  with  the  Coax  Machinegun  from 
Gunner’s  Station 

Clear  a  Machinegun  to  Prevent  Accidental  Discharge 
Operate  the  Over-Pressurization  System 
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Prepare  Vehicle  for  Fording  Operations 
TANK  COMMANDER 

Recover  by  Similar  Vehicle 
Remove/ Ins tal 1  Track  Block(s) 

Perform  Before-Operation  Checks  and  Services 
Perform  During-Operation  Checks  and  Services 
Perform  After-Operation  Checks  and  Services 
Troubleshoot  the  Gas  Particulate  Filter  Unit 
Perform  Tank  Commander's  Preventive  Maintenance 
Prepare-to-f ire  Checks  and  Services 
Perform  Tank  Commander’s  Preventive  Maintenance  After¬ 
fire  Checks  and  Services 
Troubleshoot  the  Fire  Control  System 
Engage  Targets  with  the  Main  Gun  from  Commander’s 
Weapon  Station 

Perform  Operator  Maintenance  on  Periscopes 
Stow  Ammunition 
Establish  Silent  Watch 

Prepare  Commander’s  Weapon  Station  for  Operation 

Secure  Commander’s  Weapon  Station 

Install/Remove  a  Caliber  .50  M2  HB  Machinegun 

Maintain  a  Caliber  .50  M2  HB  Machinegun 

Boresight  a  Caliber  .50  M2  HB  Machinegun 

Zero  a  Caliber  .50  M2  HB  Machinegun 

Clear  a  Caliber  .50  M2  HB  Machinegun  to  Prevent 
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Accidental  Discharge 


Engage  Targets  with  a  Caliber  .50  M2  HB  Machinegun 
Prepare  Tank  Commander’s  Weapon  for  Travel 
Evacuate  a  Wounded  Crewman 
Extinguish  a  Fire 
Replace  a  Thrown  Track 

Operate  the  Gas  Particulate  Filter  Unit 
Direct  Main  Gun  Engagements 
Direct  Machinegun  Engagements 

Perform  Manual  Extraction  of  a  Main  Gun  Round 
Boresight  and  System  Calibrate 

Perform  Operator  Before  Operations  Checks  and  Services 
NBC  System 

Inspect  Main  Gun  Ammunition  for  Serviceability 
Stow  Main  Gun  Ammunition 
Clean/Service  the  Main  Gun 
Inspect  the  Hydraulics 
Place  the  NBC  System  into  Operation 
Adjust  Crew  Compartment  Pressurization 
Perform  Gunner’s  Preventive  Maintenance  Prepare-to- 
fire  Checks  and  Services 
Perform  Gunner’s  Preventive  Maintenance  After- 
fire  Checks  and  Services 
Troubleshoot  the  Main  Gun 

Adjust  the  Operating  Cam  on  the  Main  Gun 
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Perform  Operator  Maintenance  on  the  Main  Gun 
Breechblock  Assembly 
Utilize  Land  Navigation  System 
Load/Unload  a  Grenade  Launcher 
Fire  a  Grenade  Launcher 

Engage  Targets  with  the  Coax  Machinegun  from  the 
Commander's  Weapon  Station 
Operate  the  Over  Pressurization  System 
Prepare  Vehicle  for  Fording  Operations 
CREW  TASKS 

Perform  Movement  Security  Using  Smoke  Systems 
Protect  Supplies  from  NBC  Contamination 
Perform  a  Nuclear  Contaminated  Area  Crossing 
Perform  a  Chemically  Contaminated  Area  Crossing 
Perform  Chem  Agent  Early  Warning  Using  Chem  Agent 
Alarm 

Decontaminate  Supplies  and  Equipment 
Perform  Assembly  Area  Activities 
Install  Thrown  Track 

Troubleshoot  the  Electrical  Power  System 
Start  Using  Slave  Cables 
Operate  the  Over  Pressure  System 
Boresight  Without  the  Pye-Watson  Device 
Borsight  Using  the  Pye-Watson  Device 
Execute  Pre/Post-Fire  Checks 
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Perform  PMCS 


Fight  Fires 

Tow  a  like  Vehicle 

Extract  a  Wounded  Crewman 

Refuel 

Stow  Ammunition 

Engage  Targets  in  the  Degraded  Mode 
Engage  Multiple  Main  Gun  Targets 
Engage  Targets  Simultaneously  w/Main  Gun  and 
Machinegun 

Engage  a  Target  from  Tank  using  Battlesight 

Engage  a  COAX  Target 

React  to  Indirect  Fire 

Prepare  for  a  Friendly  Nuclear  Strike 

Employ  Physical  Security  Measures 

Employ  Electronic  Counter  Countermeasures 

Prepare  a  Sketch  Range  Card 

Engage  Targets  from  a  Sketch  Range  Card  Position 

Employ  Signal  Security  Measures 

Take  Countersurveillance  Measures 

Perform  Actions  Before  an  NBC  Attack 

Perform  Actions  During  an  NBC  Attack 

Perform  Actions  After  an  NBC  Attack 

Perform  Actions  When  Operating  in  Areas  of  Residual 
Rad 
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Remove/Instal 1  Track  Block(s) 

Verify  Calibration  (Boresight) 

Troubleshoot  the  Fire  Control  System 
Troubleshoot  the  Over  Pressure  System 
Prepare  Vehicle  for  Operations 
Secure  Vehicle  for  Operations 

Perform  Operator  Maint  on  Breechblock  Assembly 
SECTION  TASKS 

Perform  Assembly  Area  Activities 
Recover  by  Similar  Vehicle 
Engage  Targets  Using  the  Two  Tank  Method 
Displace  to  Subsequent  Battle  Position 
Occupy  Covered  and  Concealed  Positions 
PLATOON  TASKS 

Protect  Supplies  from  NBC  Contamination 
Perform  a  Nuclear  Contaminated  Area  Crossing 
Perform  a  Chemically  Contaminated  Area  Crossing 
Perform  Chem  Agent  Early  Warning  Using  Chem  Agent 
Alarm 

Perform  Refueling  Activities 
Perform  Rearming  Activities 
Execute  a  Hasty  Attack 

Perform  Surveillance  (Visual  and  Electronic) 
Activities 

Move  in  a  Built-Up  Area 
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Establish  a  Platoon  Hot  Loop 

Recover  by  Similar  Vehicle 

Break  Contact  from  the  Enemy 

Engage  Targets  Using  the  Two  Tank  Method 

Occupy  a  Battle  Position  in  a  Built-Up  Area 

Execute  Actions  on  Contact 

Perform  Voice  Radio  Activities 

Execute  Actions  at  a  Halt 

Perform  a  Tactical  Road  March 

Execute  Traveling  Overwatch 

Execute  Traveling 

Execute  Bounding  Overwatch 

Defend  Against  Air  Attack 

Perform  Consolidation  and  Reorganization  Activities 

Improve  Battle  Positions 

Prepare  Subsequent  Battle  Positions 

Occupy  Battle  Positions 

Displace  to  Subsequent  Battle  Position 
Establish  Wire  Communications 
Assault  an  Enemy  Position 
Establish  All-Round  Security 
Camouflage  Vehicles  and  Equipment 
Perform  Intelligence  Gathering  Activities 
React  to  Indirect  Fire 

Prepare  for  a  Friendly  Nuclear  Strike 
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Employ  Physical  Security  Measures 

Employ  Electronic  Counter  Countermeasures 

Perform  Rupture  Force  Activities 

Perform  Reserve  Force  Activities 

Perform  Rear  Guard  Activities 

Perform  Breaching  Force  Activities 

Perform  Assault  Force  Activities 

Perform  Support  Force  Activities 

Perform  Landing  Zone  Activities 

Move  Using  a  Column  Formation 

Move  Using  a  Combat  Wedge  Formation 

Move  Using  a  Line  Formation 

Perform  Actions  at  a  Defile 

Employ  Signal  Security  Measures 

Take  Countersurveillance  Measures 

Perform  Reconnaissance  within  a  Battle  Position 

Perform  Movement  Security  Using  Smoke  Systems 

Perform  Stand-To  Activities 

Perform  Actions  Before  an  NBC  Attack 

Perform  Actions  During  an  NBC  Attack 

Perform  Actions  After  an  NBC  Attack 

Perform  Actions  When  Operating  in  Areas  of  Residual 
Rad 
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TRAINING  EFFECTIVENESS  RATING  QUESTIONNAIRE 


NOTE:  Your  comments  on  this  questionnaire  are  very 
important.  Please  complete  the  last  page  of  the 
questionnaire  when  you  have  completed  all  the  pairwise 
comparisons . 

I.  INTRODUCTION.  This  questionnaire  is  designed  to  obtain 
your  judgment  concerning  the  e i i ect i veness  of  four 
training  alternatives  with  respect  to  a  particular  task 
under  consideration.  Each  training  alternative  will  be 
compared  against  each  other  for  its  contribution  to  one  of 
six  training  effectiveness  evaluation  criteria  (this 
process  is  called  pairwise  comparison).  The  contribution 
scores  you  establish  will  then  be  entered  into  a  computer 
program  called  TOPSIS  (Technique  for  Order  Preference  by 
Similarity  to  the  Ideal  Solution).  This  program  will 
compute  your  ranking  of  the  four  alternatives  for  each  of 
the  eight  tasks  under  evaluation. 

II.  THE  TRAINING  EFFECTIVENESS  CRITERIA.  Before  you 
establish  contribution  scores  for  each  of  the  training 
alternatives  with  respect  to  the  training  effectiveness 
criteria,  let  me  first  explain  the  six  criteria.  The 
criteria  are:  stimuli  presentation,  response  presentation, 
feedback  presentation,  task  commonality,  physical 
similarity,  and  training  time.  These  criteria  are 
explained  below. 

Stimuli  presentation  (Cl)  is  an  evaluation  criteria 
used  to  judge  the  potential  the  four  alternatives  have  to 
present  information  for  the  particular  task  under 
consideration.  Based  upon  your  knowledge  of  the  gunner’s 
station  of  the  Future  Armored  Combat  System  and  your 
knowledge  of  predecessor  systems,  determine  what  type  of 
stimuli  is  required  to  perform  the  task  under 
consideration.  The  types  of  stimuli  are  visual  cues, 
audio  cues,  tactical  (sense  of  touch)  cues,  external 
stimulus  motion  cues,  internal  stimulus  motion  cues, 
olfactile  (sense  of  smell)  cues,  and  gustatile  (sense  of 
taste)  cues.  The  analyst  then  Judges  the  extent  to  which 
a  particular  training  alternative  can  provide  the  required 
stimuli  to  perform  the  task. 

The  response  presentation  criteria  (C2)  is  used  to 
evaluate  the  extent  to  which  the  training  alternatives 
permit  a  response  to  the  information  presented  to  the 
trainee  or  crewmember  for  the  particular  task  under 
review.  In  this  case,  first  determine  what  type  of 
response  is  required  of  the  operator  to  perform  the  task 
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under  consideration.  This  response  can  take  one  of  the 
following  forms:  verbal;  written;  manipulative  act; 
tracking;  or  procedural  performance.  Then  judge  the 
extent  to  which  a  particular  training  alternative  can 
accept  that  type  of  response. 

The  feedback  presentation  criteria  (C3)  is  used  to 
evaluate  the  extent  to  which  the  training  alternatives  can 
provide  feedback  to  the  operator.  Again,  first  determine 
the  type  of  feedback  required  to  reinforce  correct 
performance  or  correct  faulty  performance  on  a  particular 
task.  This  determination  is  made  based  upon  your 
knowledge  of:  the  gunner’s  station  of  the  Future  Armored 
Combat  System;  the  gunner's  station  of  predecessor 
systems;  and  the  four  training  alternatives.  Feedback 
presentation  takes  one  of  the  following  forms:  visual; 
aural;  written;  face-to-face  communication;  indirect 
communication;  tactile  (sense  of  touch);  kinesthetic 
(sense  of  motion);  olfactile  (sense  of  smell);  and 
gustatile  (sense  of  taste).  Then  Judge  the  extent  to 
which  a  particular  training  alternative  can  provide  the 
required  feedback. 

The  task  commonality  criteria  (C4)  is  used  to 
evaluate  the  extent  to  which  the  training  alternatives  can 
train  the  subtasks  (basic  steps)  of  a  particular  task. 

The  subtasks  of  a  particular  task  are  either  provided  to 
you  or  you  make  an  estimate  of  what  the  subtasks  might  be 
based  upon  your  knowledge  of  the  developing  system  and  its 
predecessor  systems  (you  will  estimate  for  this 
questionnaire) .  Then  estimate  the  percentage  of  subtasks 
a  particular  training  alternative  would  be  able  to  train 
so  that  you  can  make  a  comparison  j udgmentbetween  the 
alternatives . 

The  physical  similarity  criteria  (C5)  is  used  to 
evaluate  the  extent  to  which  the  representation  of  the 
displays  and  controls  of  the  four  training  alternatives 
are  similar  to  the  displays  and  controls  of  the  Future 
Armored  Combat  System  for  a  particular  task  under 
consideration .  Since  little  is  known  of  the  developing 
system  at  this  early  stage  of  its  developmental  cycle  you 
must  determine  if  the  particular  training  alternative  has 
the  potential  to  provide  display  and  control  physical 
(includes  fidelity)  similarity  to  what  will  likely  be 
found  in  the  gunner’s  station  of  the  Future  Armored  Combat 
System. 
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The  final  criteria  used  to  evaluate  the  effectiveness 
of  a  training  alternative  for  a  particular  task  is 
training  time  (C8) .  This  criteria  is  used  to  evaluate  the 
length  of  training  time  required  to  train  a  particular 
task  using  one  of  the  four  training  alternatives.  It  is 
not  measured  in  absolute  time  (something  a  later  field 
test  would  establish)  but  rather  reflects  a  comparison 
among  alternatives. 

III.  THE  RELATIVE  IMPORTANCE  OF  THE  SIX  CRITERIA.  Now  we 
must  establish  the  degree  to  which  each  criteria 
contributes  to  the  goal  of  selecting  the  most  effective 
training  alternative  for  the  task  under  consideration. 

This  establishes  what  is  called  the  decision  maker’s 
weights  (you  are  the  decision  maker  in  this  case) .  For 
example,  if  you  believe  that  task  commonality  is  twice  as 
important  as  training  time  then  the  comparison  ratings  you 
assign  for  task  commonality  should  receive  twice  the 
credit  in  the  rating  process.  To  obtain  these  weights  you 
will  compare  each  criteria  with  another  criteria  using  the 
rating  scale  below: 

THE  PAIRWISE  COMPARISON  SCALE 


Intensity  of  Definition  Explanation 

Importance 


1 

Equal  importance  of 
both  elements 

Two  elements  contribute 
equally  to  training 
ef  f ectiveness 

3 

Weak  importance  of 
one  element  over 
the  other 

Experience  and  judgement 
slightly  favor  one 
criteria  over  the  other 

5 

Essential  or  strong 
importance  of  one 
element  over  the 
other 

Experience  and  judgement 
strongly  favor  one 
criteria  over  the  other 

7 

Demonstrated 
importance  of  one 
element  over  the 
other 

A  criteria  is  strongly 
favored  and  its  dominance 
is  demonstrated  in 
practice 

9 

Absolute  importance 
of  one  element  over 
the  other 

The  evidence  favoring  one 
criteria  over  another  is 
of  the  highest  possible 
order  of  affirmation 

2,4, 

Intermediate  values 

Compromise  is  needed 
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6,8  between  two  adjacent  between  two  judgements 

j  udgements 

Reciprocals 

If  the  second  item  of  comparison  is  the  stronger 
contributor  to  training  effectiveness  then  the 
score  given  in  the  comparison  rating  is  the 
reciprocal  value  of  the  appropriate  rating  in  the 
scale  above  (e.g.  1/3 , 1/5 , 1/6)  . 

Here  is  an  example  of  how  to  use  this  scale.  The 
first  comparison  to  make  is  to  establish  the  contribution 
stimuli  presentation  makes  to  training  effectiveness  when 
compared  to  response  presentation.  Let  us  say  you  decide 
that  the  ability  of  any  training  alternative  to  present 
information  (stimuli  presentation)  is  strongly  more 
important  to  the  goal  of  achieving  an  effective  training 
system  than  the  ability  of  any  training  alternative  to 
permit  a  trainee  response  to  information  (response 
presentation).  The  rating  scale  rates  a  'strongly  more 
important  (strongly  favor)*  comparison  a  5.  Therefore, 
you  would  rate  the  stimuli  presentation  versus  response 
presentation  a  5.  Note:  The  order  of  comparison  is 
important.  For  example,  if  you  feel  that  response 
presentation  is  strongly  more  important  than  stimuli 
presentation  you  would  rate  the  stimuli  presentation 
versus  response  presentation  its  reciprocal  value  or  1/5. 

Now,  please  answer  the  following  questions  using  the 
1  to  9  rating  scale  (or  reciprocal  value) . 

1.  To  what  degree  does  stimuli  presentation  contribute  to 
training  effectiveness  when  compared  to  response 
presentation? 


2.  To  what  degree  does  stimuli  presentation  contribute  to 
training  effectiveness  when  compared  to  feedback 
presentation? 


3.  To  what  degree  does  stimuli  presentation  contribute  to 
training  effectiveness  when  compared  to  task  commonality? 


4.  To  what  degree  does  stimuli  presentation  contribute  to 
training  effectiveness  when  compared  to  physical 
similarity? 


5.  To  what  degree  does  stimuli  presentation  contribute  to 
training  effectiveness  when  compared  to  training  time'? 
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6.  To  what  degree  does  response  presentation  contribute 
to  training  effectiveness  when  compared  to  feedback 
presentation? 


7.  To  what  degree  does  response  presentation  contribute 
to  training  effectiveness  when  compared  to  task 
commonal i ty ? 


8.  To  what  degree  does  response  presentation  contribute 
to  training  effectiveness  when  compared  to  physical 
similarity? 


9.  To  what  degree  does  response  presentation  contribute 
to  training  effectiveness  when  compared  to  training  time? 


10.  To  what  degree  does  feedback  presentation  contribute 
to  trailing  effectiveness  when  compared  to  task 
commonal i ty ? 


11.  To  what  degree  does  feedback  presentation  contribute 
to  training  effectiveness  when  compared  to  physical 
similarity? 


12.  To  what  degree  does  feedback  presentation  contribute 
to  training  effectiveness  when  compared  to  training  time? 


13.  To  what  degree  does  task  commonality  contribute  to 
training  effectiveness  when  compared  to  physical 
similarity? 


14.  To  what  degree  does  task  commonality  contribute  to 
training  effectiveness  when  compared  to  training  time? 


15.  To  what  degree  does  physical  similarity  contribute  to 
training  effectiveness  when  compared  to  training  time? 


IV.  TRAINING  ALTERNATIVES.  Now  that  we  have  established 
your  ranking  of  the  six  criteria  with  respect  to  their 
contribution  to  training  effectiveness  it  is  time  to 
introduce  the  four  training  alternatives.  The  four 
training  alternatives  are:  classroom  or  unit  instructor 
based  instruction,  part  task  computer  based  instruction, 
full  function  stand  alone  trainers,  amd  embedded  training. 

Classroom  or  unit  instructor  based  instruction  (Al) 
is  the  traditional  training  given  by  an  instructor.  Both 
small  group  instruction  (anywhere  from  1:6  to  1:16 
instructor  to  student  ratio)  and  large  group  instruction 
(1:17  to  1:60  instructor  to  student  ratio)  are  included  in 
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this  category.  Instructors  provide  instruction  through 
traditional  audio-visual  means.  This  includes 
blackboards,  overhead  projectors,  35mm  slide  projectors, 
films,  video  tapes,  guest  speakers  and  panel 
discussions.  Student  performance  evaluation  ranges  from 
instructor  subjective  opinion  of  classroom  contribution 
through  practical  exercises  to  graded  individual  papers 
and  written  or  hands  on  (hands  on  the  actual  piece  of 
equipment)  performance  examinations. 

Part  task,  computer  based  instruction  (A2)  ranges 
from  instruction  provided  by  a  table  top  personal  computer 
through  a  learning  station  linked  to  a  mainframe  computer 
to  a  table  top  computer  based  delivery  system  specifically 
designed  for  training.  This  latter  subcategory  includes 
systems  such  as  the  Electronic  Information  Delivery  System 
(which  provides  enhanced  learning  through  a  touch  screen 
display  vice  a  pure  keyboard  operation)  and  the  Video  Disc 
Gunnery  Simulator  (a  table  top  videodisc  based  trainer 
designed  to  familiarize  a  tank  gunner  with  the  fire 
control  equipment  at  his  crew  station  and  train  the  tasks 
engage  the  target  with  the  main  gun  /coax  machinegun  from 
the  gunner’s  station  under  varing  combat  conditions). 
Instructors  or  facilitators  are  present  for  the 
instruction  but  only  provide  answers  to  specific  student 
questions  on  lesson  content,  operation  of  the  computer 
providing  the  instruction,  or  direct  movement  to  the  next 
training  step  (when  the  computer  cannot  perform  this 
function) .  This  category  can  include  collective  trainers 
whose  function  is  to  train  partial  task  groupings  or  parts 
of  individual  tasks  without  fully  replicating  the  actual 
piece  of  equipment  or  crewmember  station.  A  good  example 
of  this  is  the  Combined  Arms  Tactical  Trainer  (often 
referred  to  as  Simulation  Networking  -  SIMNET) .  This 
trainer  trains  crews  in  collective  gunnery  and  tactical 
tasks  by  providing  only  the  required  fidelity  necessary  to 
train  the  tasks  chosen  for  training. 

Full  function  stand  alone  trainers  (A3)  are  designed 
to  replicate  performance  on  a  task  in  the  same  way  as  it 
is  performed  on  the  operational  equipment.  They  are  also 
designed  to  replicate  the  physical  environment  of  the 
crewmember's  station  (knobs,  controls,  switches,  seats, 
legroom,  etc.).  Good  examples  of  these  type  of  trainers 
are  flight  simulators  for  helicopters  and  fixed  wing 
aircraft  and  the  Unit  Conduct  of  Fire  Trainer  (UCOFT)  for 
the  tank  or  Bradley  Fighting  Vehicle  (BFV) .  The  UCOFT 
replicates  the  physical  characteristics  of  the  vehicle 
commander  and  gunner  positions  and  uses  a  computer  driven 
matrix  of  685  increasingly  difficult  exercises  to  train 
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vehicle  commander  and  gunner  teams  on  target  engagement 
techniques.  These  stand  alone  trainers  are  frequen  tly 
quite  similar  to  their  embedded  training  counterparts  with 
one  noteable  exception.  It  is  usually  not  possible  to 
replicate  the  conditions  of  weather  which  would  affect 
human  performance  on  a  task.  For  example,  flight 
simulators  and  the  UCOFT  are  enclosed  in  environmentally 
controlled  rooms  or  shelters,  so  the  temperature  and 
humididty  which  may  effect  a  tank  crew’s  performance  on  a 
task  outdoors  is  not  replicated  in  the  simulator.  These 
full  funtion  stand  alone  trainers  often  have  limited 
accessibility  since  they  would  not  be  produced  in  the  same 
quantities  as  would  the  operational  equipment  (a  function 
of  cost) .  Instructors  or  operators  are  present  for  the 
training  but  facilitate  the  students  progress  through  the 
training  software  rather  than  provide  the  training 
themselves.  These  trainers  would  either  provide  their  own 
performance  evaluation  and  feedback  or  would  be 
supplemented  by  an  instructor  or  operator  to  provide  any 
additional  required  evaluation  or  feedback  on  task 
performance . 

Embedded  training  (A4)  is  the  fourth  alternative. 
Embedded  training  is  training  that  is  provided  by 
capabilities  designed  to  be  built  into  or  added  onto 
operational  systems  to  enhance  and  maintain  the  skill 
proficiency  necessary  to  operate  and  maintain  that 
equipment  end  item  (U.S.  Army  Policy  Letter,  3  March 
1987).  Embedded  training  subsystems  lie  on  a  continuum 
ranging  from  an  appended  training  subsystem  to  subsystems 
totally  integrated  with  the  operational  equipment. 

Appended  subsystems  can  be  quickly  attached  to  existing 
mounting  hardware  and  data  or  electronic  connections.  At 
the  mid-point  are  training  subsystems  which  are 
permanently  mounted  to  the  combat  system  but  are  an 
adjunct  to  the  operational  hardware.  On  the  opposite  end 
of  this  continuum  are  training  subsystems  which  are 
totally  integrated  into  the  operational  hardware  (i.e., 
subsystems  which  share  the  same  black  box) .  Embedded 
training  must  provide,  in  some  fashion,  performance 
assessment  to  determine  the  proficiency  level  of  the  user, 
feedback  to  the  user  to  improve  performance  or  reinforce 
correct  performance,  and  record  keeping  to  manage  the 
training  proficiency  progression  of  the  user. 

Instructors  or  facilitators  can  be  present  for  the  task 
training  provided  by  an  embedded  training  capability  but 
do  not  present  the  training.  They  perform  the  same  role 
as  in  alternatives  two  and  three.  Feedback  may  be 
provided  by  the  embedded  training  capability  itself  or  by 
an  instructor  or  facilitator  designated  to  observe  the 
training . 
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V.  TASK  RATINGS. 


A.  PERFORM  BEFORE  OPERATIONS  CHECKS  AND  SERVICES. 
This  is  the  first  task  for  which  you  will  establish 
comparison  ratings.  Answer  the  following  questions 
keeping  in  mind  what  you  know  about  the  requirements  of 
the  M1A1  Abrams  tank  and  the  proposed  capabilites  and 
characteristics  of  the  Future  Armored  Combat  System  with 
respect  to  this  particular  task.  This  step  evaluates  the 
potential  each  training  alternative  has  in  training  the 
perform  before  operations  checks  and  services  task  for 
each  of  the  six  measures  of  training  effectiveness 
(criteria)  we  have  established.  Note:  Use  the  previous 
rating  scale  except  in  this  case  we  are  comparing  the 
training  alternatives  just  discussed  instead  of  the 
effectiveness  criteria.  Please  answer  the  following 
questions . 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
part  task  computer  based  instruction  (A2)?  Cl-9,1/2-1/9] 


2,  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
full  function  stand  alone  trainers  (A3)?  Cl-9,1/2-1/9] 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
embedded  training  (A4)?  Cl-9,1/2-1/9] 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  full 
function  stand  alone  trainers  (A3)?  Cl-9,1/2-1/9] 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  embedded 
training  (A4)?  [ 1-9 , 1/2- 1/9 3 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  embedded 
training  (A4)?  C 1-9 , 1/2- 1/9  ] 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  perform  before  operations  checks  and  services 
when  compared  to  part  task  computer  based  instruction 
(A2) ?  [ 1-9 , 1/2-1/9] 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  perform  before  operations  checks  and  services 
when  compared  to  full  function  stand  alone  trainers  (A3) 7 
[1-9, 1/2-1/9] 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  perform  before  operations  checks  and  services 
when  compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
perform  before  operations  checks  and  services  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

[1-9, 1/2-1/93 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
perform  before  operations  checks  and  services  when 
compared  to  embedded  training  (A4)?  [1  9,1/2-1/93 
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12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
embedded  training  (A4) ?  [  1-9 . 1/2-1/9] 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  part  task 
computer  based  instruction  (A2)?  C 1  —  9 , 1/2—1/93 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/91 


13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  embedded 
training  (A4) ?  [1-9,1/2-1/91 


Id.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [1-9,1/2-1/91 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  embedded  training 
(A4) ?  [1-9,1/2-1/93 


18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  embedded  training 
(A4)?  [1-9,1/2-1/91 


TASK  COMMONALITY. 


19.  To  what  degree  or  level  o f  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  perform  before  operations  checks  and  services  when 
compared  to  part  task  computer  based  instruction  (A2)'? 

C 1-9 . 1/2-1/9] 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  perform  before  operations  checks  and  services  when 
compared  to  full  function  stand  alone  trainers  (A3)  ? 

C 1-9, 1/2-1/93 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  perform  before  operations  checks  and  services  when 
compared  to  embedded  training  (A4)?  [ 1-9 , 1/2- 1/9 ] 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
full  function  stand  alone  trainers  (A3)  °  [19,1/2-1/9] 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  perform 
before  operations  checks  and  services  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


PHYSICAL  SIMILARITY. 
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25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  part  task 
computer  based  instruction  (A2)?  Cl-9, 1/2-1/9] 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  a  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  full 
function  stand  alone  trainers  (A3) 7  C 1-9 , 1/2- 1/9  ] 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  a  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  perform  before 
operations  checks  and  services  when  compared  to  embedded 
training  (A4) ?  C 1-9. 1/2-1/9] 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  a 
physically  similar  displays,  controls,  or  equipment 
necessary  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  full  function  stand 
alone  trainers  (A3)?  C 1-9 , 1/2- 1/9 ] 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  a 
physically  similar  displays,  controls,  or  equipment 
necessary  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  embedded  training 
(A4)?  [ 1-9 , 1/2- 1/9  ] 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  a  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  perform  before  operations  checks  and 
services  when  compared  to  embedded  training  (A4)? 

Cl-9 , 1/2-1/93 


TRAINING  TIME. 


B 


12 


31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  part  task  computer 
based  instruction  (A2)?  1 1 -9 , 1 /2- 1/9 ] 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  full  function  stand 
alone  trainers  (A3)?  C 1 -9 , 1 / 2- 1 / 9  ] 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  perform  before  operations 
checks  and  services  when  compared  to  embedded  training 
(A4)?  [1-9, 1/2-1/fl] 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  perform  before  operations  checks  and 
services  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1-9 , 1/2-1/93 


35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  perform  before  operations  checks  and 
services  when  compared  to  embedded  training  (A4)? 

C 1-9 , 1/2-1/93 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  perform  before  operations  checks  and 
services  when  compared  to  embedded  training  (A4)? 
[1-9,1/2-1/93 


B.  TROUBLESHOOT  THE  FIRE  CONTROL  SYSTEM. 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to  part 
task  computer  based  instruction  (A2)?  [1-9,1/2-1/93 
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2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9] 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4) ?  C 1—9 , 1/2-1/93 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  full  function  stand 
alone  trainers  (A3)?  C 1—9, 1/2-1/91 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  present 
information  required  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  embedded  training 
( A4 ) ?  [1-9,1/2-1/93 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  embedded  training 
( A4 ) ?  [1-9,1/2-1/93 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  baaed  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  troubleshoot  the  fire  control  system  when 
compared  to  part  task  computer  based  instruction  (A2)? 
[1-9, 1/2-1/93 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
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the  task  troubleshoot  the  fire  control  system  when 
compared  to  full  function  stand  alone  trainers  (A3)? 
C 1-9 , 1/2-1/93 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  troubleshoot  the  fire  control  system  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  par*,  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9 ] 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4)?  [1-9,1/2-1/93 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  troubleshoot 
the  fire  control  system  when  compared  to  part  task 
computer  based  instruction  (A2) ?  [1-9,1/2-1/93 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  troubleshoot 
the  fire  control  system  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 
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15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  troubleshoot 
the  fire  control  system  when  compared  to  embedded  training 
(A4) ?  [1-9,1/2-1/93 


16.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  troubleshoot  the  fire  control 
system  when  compared  to  full  function  stand  alone  trainers 
(A3)?  [1-9,1/2-1/93 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  troubleshoot  the  fire  control 
system  when  compared  to  embedded  training  (A4)? 

[1-9 , 1/2-1/93 


18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  troubleshoot  the  fire  control 
system  when  compared  to  embedded  training  (A4) ? 

[ 1-9 , 1/2-1/93 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  troubleshoot  the  fire  control  system  when  compared  to 
part  task  computer  based  instruction  (A2)?  [1-9,1/2-1/93 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  troubleshoot  the  fire  control  system  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4)?  [ 1-9 , 1/2- 1/9  ] 
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22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9 3 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task 
troubleshoot  the  fire  control  system  when  compared  to 
embedded  training  (A4) ?  C 1-9 , 1/2-  1/9  ] 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  part  task  computer 
based  instruction  (A2) ?  [ 1-9 , 1/2- 1/9 ] 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [1-9,1/2-1/93 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  troubleshoot  the 
fire  control  system  when  compared  to  embedded  training 
(A4) ?  [1-9,1/2-1/93 
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28.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  troubleshoot  the  fire  control  system  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

C 1-9 , 1/2-1/9] 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  troubleshoot  the  fire  control  system  when 
compared  to  embedded  training  (A4) ?  [ 1-9 , 1/2- 1/9  ] 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  troubleshoot  the  fire  control  system  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  troubleshoot  the  fire 
control  system  when  compared  to  part  task  computer  based 
instruction  (A2) ?  [1-9,1/2-1/93 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  troubleshoot  the  fire 
control  system  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  troubleshoot  the  fire 
control  system  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/93 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  troubleshoot  the  fire  control  system 
when  compared  to  full  function  stand  alone  trainers  (A3)? 
[1-9 , 1/2-1/93 
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35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  troubleshoot  the  fire  control  system 
when  compared  to  embedded  training  (A4) ?  C 1-9 , 1/2- 1/9 3 


38.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  troubleshoot  the  fire  control  system  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


C.  EVACUATE  A  WOUNDED  CREWMAN. 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  part  task  computer  based 
instruction  (A2)  ?  [1-9,1/2-1/93 


2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  embedded  training  (A4)? 
[1-9 , 1/2-1/93 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  embedded  training  (A4)? 

[1-9 , 1/2-1/93 
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0.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
Information  required  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  embedded  training  (A4)? 

1 1-9 , 1/2- 1/9  ] 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  evacuate  a  wounded  crewman  when  compared  to  part 
task  computer  based  instruction  (A2)?  [ 1-9 , 1/2- 1/9  ] 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  evacuate  a  wounded  crewman  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  evacuate  a  wounded  crewman  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
evacuate  a  wounded  crewman  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
evacuate  a  wounded  crewman  when  compared  to  embedded 
training  (A4)?  C 1-9 , 1/2- 1/9 3 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task 
evacuate  a  wounded  crewman  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 
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FEEDBACK  PRESENTATION 


13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  part  task  computer  based 
instruction  (A2)?  C 1-9 , 1/2-1/9  ] 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/03 


15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  evacuate  a 
wounded  crewman  when  compared  to  embedded  training  (A4)? 
[1-9,1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  evacuate  a  wounded  crewman  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

[1-9 , 1/2-1/93 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  evacuate  a  wounded  crewman  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 


18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  evacuate  a  wounded  crewman  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  evacuate  a  wounded  crewman  when  compared  to  part  task 
computer  based  instruction  (A2)?  [1-9,1/2-1/93 
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20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  evacuate  a  wounded  crewman  when  compared  to  full 
function  stand  alone  trainers  (A3)?  C 1-9 , 1/2-  1/9  3 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  evacuate  a  wounded  crewman  when  compared  to  embedded 
training  (A4) ?  C 1-9 , 1/2-  1/9  ] 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  evacuate 
a  wounded  crewman  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [1-9,1/2-1/93 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  cover  each  of  the 
subtaaks  (basic  steps)  required  to  train  the  task  evacuate 
a  wounded  crewman  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/93 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  evacuate 
a  wounded  crewman  when  compared  to  embedded  training  (A4)? 
[ 1-9 , 1/2-1/93 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  part  task  computer  based 
instruction  (A2)?  [1-9,1/2-1/93 


29.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
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equipment  necessary  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [ 1-9 , 1/2-1/9 ] 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  evacuate  a  wounded 
crewman  when  compared  to  embedded  training  (A4) ? 

C 1-0 , 1/2- 1/0] 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  evacuate  a  wounded  crewman  when  compared  to 
full  function  stand  ilone  trainers  (A3)?  [ 1-9 , 1/2- 1/9  ] 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  evacuate  a  wounded  crewman  when  compared  to 
embedded  training  (A4) ?  C 1-9 , 1/2- 1/9  ] 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  evacuate  a  wounded  crewman  when  compared  to 
embedded  training  (A4) ?  C 1-9 , 1/2- 1/9 ] 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  evacuate  a  wounded  crewman 
when  compared  to  part  task  computer  based  instruction 
(A2) ?  [1-9,1/2-1/93 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  evacuate  a  wounded  crewman 
when  compared  to  full  function  stand  alone  trainers  (A3)? 
[1-9,1/2-1/93 
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33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  evacuate  a  wounded  crewman 
when  compared  to  embedded  training  (A4)  ?  [ 1-9 , 1/2- 1/9 ] 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  evacuate  a  wounded  crewman  when  compared 
to  full  function  stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9  ] 


35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  evacuate  a  wounded  crewman  when  compared 
to  embedded  training  (A4)?  C 1-9  ,  1/2- 1/9 ] 


36.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  evacuate  a  wounded  crewman  when  compared  to 
embedded  training  (A4) 0  C 1-9 , 1/2- 1/9  ] 


D.  REPLACE  A  THROWN  TRACK. 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  part  task  computer  based 
instruction  (A2)7  C 1-9 , 1/2-  1/9 3 


2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1—9 . 1/2—1/91 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  embedded  training  (A4)? 

C 1-9, 1/2-1/93 
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4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  replace  a  thrown 
track  when  compared  to  full  function  stand  alone  trainers 
(A3) ?  [1-9,1/2-1/93 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  replace  a  thrown 
track  when  compared  to  embedded  training  (A4)'? 

[1-9 , 1/2-1/93 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  replace  a  thrown 
track  when  compared  to  embedded  training  (A4) ? 

[1-9 ,1/2-1/93 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  replace  a  thrown  track  when  compared  to  part  task 
computer  based  instruction  (A2) ?  [1-9,1/2-1/93 


0.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  replace  a  thrown  track  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  replace  a  thrown  track  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
replace  a  thrown  track  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 
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11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
replace  a  thrown  track  when  compared  to  embedded  training 
(A4)?  [1-9,1/2-1/93 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  replace 
a  thrown  track  when  compared  to  embedded  training  ( A4 )  ? 

C 1-9 , 1/2-  1/9  3 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  part  task  computer  based 
instruction  (A2) ?  [ 1-9 , 1/2- 1/9 ] 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1-9 . 1/2- 1/9  ] 


13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  replace  a 
thrown  track  when  compared  to  embedded  training  (A4)? 
i 1 -9 , 1/2-1/9] 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  replace  a  thrown  track  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

C 1-9 , 1/2-1/9] 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  replace  a  thrown  track  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 
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18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  replace  a  thrown  track  when 
compared  to  embedded  training  (A4)?  C 1-9 , 1/2- 1/9 ] 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  replace  a  thrown  track  when  compared  to  part  task 
computer  based  instruction  (A2) ?  [ 1-9 , 1/2- 1/9  ] 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  replace  a  thrown  track  when  compared  to  full  function 
stand  alone  trainers  (A3)?  C 1-9 , 1/2-1/93 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  replace  a  thrown  track  when  compared  to  embedded 
training  (A4) ?  C 1-9 , 1/2- 1/9  ] 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  replace 
a  thrown  track  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1  —  9 , 1/2— 1/9] 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  replace 
a  thrown  track  when  compared  to  embedded  training  (A4)? 
[1-9, 1/2-1/93 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  replace 
a  thrown  track  when  compared  to  embedded  training  (A4) ? 

C 1 -9 , 1/2-1/93 


B  -  27 


PHYSICAL  SIMILARITY. 


25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  replace  a  thrown 
track  when  compared  to  part  task  computer  based 
instruction  (A2)?  C 1-9 , 1/2- 1/9  ] 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  replace  a  thrown 
track  when  compared  to  full  function  stand  alone  trainers 
(A3)?  [1-9,1/2-1/93 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  replace  a  thrown 
track  when  compared  to  embedded  training  (A4) 

[1-9 , 1/2-1/93 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  replace  a  thrown  track  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  replace  a  thrown  track  when  compared  to 
embedded  training  (A4)  ?  t 1-9 ,  1/2-1/93 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  replace  a  thrown  track  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


TRAINING  TIME. 
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31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  replace  a  thrown  track 
when  compared  to  part  task  computer  based  instruction 
(A2) ?  [1-9,1/2-1/93 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  replace  a  thrown  track 
when  compared  to  full  function  stand  alone  trainers  (A3)  ? 
[1-9 , 1/2-1/93 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  replace  a  thrown  track 
when  compared  to  embedded  training  ( A4 ) ?  [1-9,1/2-1/93 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  require  less  training  time 
to  train  the  task  replace  a  thrown  track  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


35.  To  what  decree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  replace  a  thrown  track  when  compared  to 
embedded  training  (A4)?  [1-9,1/2-1/93 


36.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  replace  a  thrown  track  when  compared  to 
embedded  training  (A4)?  [1-9,1/2-1/93 


E.  ENGAGE  TARGETS  WITH  THE  MAIN  GUN  FROM  THE  GUNNER’S 
STATION. 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner's  station  when 
compared  to  part  task  computer  based  instruction  (A2) ? 
[1-9, 1/2-1/93 
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2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner’s  station  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

C 1-9 , 1/2- 1/9  ] 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner’s  station  when 
compared  to  embedded  training  (A4) ?  [ 1-9 , 1 /2- 1/9 ] 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner’s  station  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9 ] 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner’s  station  when  compared  to 
embedded  training  (A4) ?  El— 9, 1/2—1/93 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner's  station  when  compared  to 
embedded  training  (A4)?  [ 1-9 , 1/2-  1/9  ] 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  part  task  computer  based 
instruction  (A2) ?  C 1  —  9, 1/2  —  1/93 


B  -  30 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner’s  station  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [ 1-9 , 1/2-  1/9  ] 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4)^ 
[1-9 , 1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
engage  targets  with  the  main  gun  from  the  gunner's  station 
when  compared  to  full  function  stand  alone  trainers  (A3)? 

[ 1-9 . 1/2-1/93 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
engage  targets  with  the  main  gun  from  the  gunner's  station 
when  compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner’s  station  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  main  gun  from  the  gunner’s  station  when  compared 
to  part  task  computer  based  instruction  (A2)? 

[ 1-9 , 1/2-1/93 
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14.  To  what  degree  or  level  of  potential  does  unit  or 
olasaroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  main  gun  from  the  gunner’s  station  when  compared 
to  full  function  stand  alone  trainers  (A3)?  C 1-9 , 1/2- 1/9 ] 


15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  main  gun  from  the  gunner’s  station  when  compared 
to  embedded  training  (A4)?  C 1-9  ,  1/2-  1/9  ] 


16.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  main 
gun  from  the  gunner's  station  when  compared  to  full 
function  stand  alone  trainers  (A3)?  C 1-9 , 1/2-1/93 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  main 
gun  from  the  gunner's  station  when  compared  to  embedded 
training  (A4) ?  [1-9,1/2-1/93 


18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  main 
gun  from  the  gunner’s  station  when  compared  to  embedded 
training  (A4/?  C 1-9 , 1/2-1/93 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  main  gun  from  the  gunner's 
station  when  compared  to  part  task  computer  based 
instruction  (A2) ?  C 1-9 , 1/2- 1/9 ] 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  main  gun  from  the  gunner’s 
station  when  compared  to  full  function  stand  alone 
trainers  (A3) ^  C 1-9  ,  1/2- 1/9  3 
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21.  To  what  degree  or  level  of  potential  does  unit  or 
claaaroom  inatructor  based  inatruction  (Al)  have  to  cover 
each  of  the  subtaaka  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  main  gun  from  the  gunner’s 
station  when  compared  to  embedded  training  (A4)? 

C 1-9 . 1/2- 1/9 ] 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner’s  station  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

[1-9 , 1/2— 1/9] 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  Instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner's  station  when 
compared  to  embedded  training  (A4) ?  [ 1-9 , 1/2- 1/9 ] 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
targets  with  the  main  gun  from  the  gunner’s  station  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner’s  station  when  compared  to 
part  task  computer  based  instruction  (A2)9  [1-9,1/2-1/93 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner’s  station  when  compared  to 
full  function  stand  alone  trainers  (A3)?  C 1-9 .  1/2-1/93 
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27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  engage  targets  with 
the  main  gun  from  the  gunner's  station  when  compared  to 
embedded  training  (A4)  ?  [1-9,1/2-1/93 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  engage  targets  with  the  main  gun  from  the 
gunner’s  station  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [1-9,1/2-1/93 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4) ? 
[1-9,1/2-1/93 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  engage  targets  with  the  main  gun  from  the 
gunner’s  station  when  compared  to  embedded  training  (A4)? 
[1-9 , 1/2-1/93 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
main  gun  from  the  gunner's  station  when  compared  to  part 
task  computer  baaed  instruction  (A2)?  [1-9,1/2-1/93 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
main  gun  from  the  gunner’s  station  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 
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33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
main  gun  from  the  gunner’s  station  when  compared  to 
embedded  training  (A4)  ?  C 1-9  ,  1/2-  1/9  ] 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  engage  targets  with  the  main  gun  from 
the  gunner's  station  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [ 1 -9 , 1 / 2-  1  / 9  ] 


35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  engage  targets  with  the  main  gun  from 
the  gunner’s  station  when  compared  to  embedded  training 
(A4) ?  [1-9,1/2-1/93 


36.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4) 
[1-9,1/2-1/93 


F.  ADJUST  CREW  COMPARTMENT  PRESSURIZATION. 

STIMULI  PRESENTATION. 

1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  part  task 
computer  based  instruction  (A2)?  [ 1-9 , 1/2- 1/9  ] 


2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  embedded 
training  (A4) ?  [1-9.1/2-1/93 


B  -  35 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  CA2)  have  to  present 
information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  full  function 
stand  alone  trainers  (A3)?  C 1-9 , 1/2- 1/9  ] 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  embedded 
training  (A4) ?  [1-9,1/2-1/93 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  adjust  crew  compartment  pressurization  when 
compared  to  part  task  computer  based  instruction  (A2) ? 

[1-9 , 1/2-1/93 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  adjust  crew  compartment  pressurization  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

[1-9, 1/2-1/93 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4)? 
[1-9 , 1/2-1/93 
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10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
adjust  crew  compartment  pressurization  when  compared  to 
full  function  stand  alone  trainers  (A3)’  C 1-9 , 1/2-1/93 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
adjust  crew  compartment  pressurization  when  compared  to 
embedded  training  (A4)  ’  [ 1-9 , 1/2- 1/9  ] 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  adjust 
crew  compartment  pressurization  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  part  task 
computer  based  instruction  (A2) ?  [ 1-9 , 1/2- 1/9 ] 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9  ] 


15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  embedded 
training  (A4) ?  [1-9, 1/2-1/93 


15.  To  what  degree  or  lev  '  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 
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17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  Instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  embedded  training  (A4)'? 
[1-9, 1/2-1/9] 


18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  embedded  training  (A4)<? 

C 1-0 , 1/2- 1/03 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  adlust  crew  compartment  pressurization  when  compared 
to  part  task  computer  based  instruction  (A2) ? 
[1-9,1/2-1/93 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  adjust  crew  compartment  pressurization  when  compared 
to  full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  adjust  crew  compartment  pressurization  when  compared 
to  embedded  training  (A4)?  [1-9,1/2-1/93 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  adjust 
crew  compartment  pressurization  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  adjust 
crew  compartment  pressurization  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 
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24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  adjust 
crew  compartment  pressur i zati on  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  part  task 
computer  based  instruction  (A2)?  [ 1-9 , 1/2- 1/9  ] 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  adjust  crew 
compartment  pressurization  when  compared  to  embedded 
training  (A4) ?  [1-9,1/2-1/93 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  adjust  crew  compartment  pressurization  when 
compared  to  full  function  stand  alone  trainers  (A3) ? 

[1-9 , 1/2-1/93 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  adjust  crew  compartment  pressurization  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/93 
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30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  adjust  crew  compartment  pressurization  when 
compared  to  embedded  training  (A4)?  [  1-9 , 1/2-  1/9  ] 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  part  task  computer  based 
instruction  (A2) ?  C 1 - 3 , 1/2- 1/9  ] 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1-9 , 1/2-1/93 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  adjust  crew  compartment 
pressurization  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/9] 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  adjust  crew  compartment  pressurization 
when  compared  to  full  function  stand  alone  trainers  (A3) ? 
[1-9 , 1/2-1/9] 


35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  adjust  crew  compartment  pressurization 
when  compared  to  embedded  training  (A4) ?  [ 1-9 , 1/2- 1/9 ] 


3d.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  adjust  crew  compartment  pressurization  when 
compared  to  embedded  training  (A4)?  C 1-9 , 1/2-1/93 


G.  ZERO  THE  MACHINEGUN 
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STIMULI  PRESENTATION. 


1.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  part  task  computer  based 
instruction  (A2)?  C 1-9 , 1/2- 1/9  ] 


2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  full  function  stand  alone 
trainers  (A3)?  Cl  —  9, 1/2— 1/9] 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  ( A4 ) ? 

[ 1-9 . 1/2-1/9] 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1-9 , 1/2- 1/9 3 


5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/93 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/93 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
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the  task  zero  the  machinegun  when  compared  to  part  task 
computer  based  instruction  (A2) ?  [ 1-9 , 1/2- 1/9  ] 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (A15  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  zero  the  machinegun  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4) ? 
[1-9 , 1/2-1/93 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task  zero 
the  machinegun  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task  zero 
the  machinegun  when  compared  to  embedded  training  (A4)7 
[1-9 , 1/2-1/93 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  zero 
the  machinegun  when  compared  to  embedded  training  (A4)  '> 
[1-9 , 1/2-1/93 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  zero  the 
machinegun  when  compared  to  part  task  computer  based 
instruction  (A2)?  [1-9,1/2-1/93 
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14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  zero  the 
machinegun  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  (A4)? 
[1-9,1/2-1/93 


16.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  zero  the  machinegun  when 
compared  to  full  function  stand  alone  trainers  (A3) ? 

[1-9 , 1/2-1/93 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  zero  the  machinegun  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


18,  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  zero  the  machinegun  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  zero  the  machinegun  when  compared  to  part  task 
computer  based  instruction  (A2)?  [1-9,1/2-1/93 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  zero  the  machinegun  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/93 
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21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  zero  the  machinegun  when  compared  to  embedded 
training  (A4)?  [1-9,1/2-1/93 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  zero  the 
machinegun  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [1-9,1/2-1/93 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  (A4)? 
[1-9,1/2-1/93 


24.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  zero  the 
machinegun  when  compared  to  embedded  training  (A4) ? 
[1-9,1/2-1/93 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  zero  the  machinegun 
when  compared  to  part  task  computer  based  instruction 
(A2) ?  Cl-9,1/2-1/93 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  zero  the  machinegun 
when  compared  to  full  function  stand  alone  trainers  (A3)? 
[1-9 , 1/2-1/93 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necessary  to  train  the  task  zero  the  machinegun 
when  compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 
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28.  To  what  degree  or  level  of  potential  doe3  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  zero  the  machinegun  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays ,  controls,  or  equipment  necessary  to 
train  the  task  zero  the  machinegun  when  compared  to 
embedded  training  (A4) ?  [1-9,1/2-1/93 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necessary  to 
train  the  task  zero  the  machinegun  when  compared  to 
embedded  training  (A4)?  [1-9,1/2-1/93 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  zero  the  machinegun  when 
compared  to  part  task  computer  based  instruction  (A2) ? 
[1-9, 1/2-1/93 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  zero  the  machinegun  when 
compared  to  full  function  stand  alone  trainers  (A3)? 

[1-9, 1/2-1/93 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  zero  the  machinegun  when 
compared  to  embedded  training  ( A4 ) ?  [1-9,1/2-1/93 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  zero  the  machinegun  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 
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35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  zero  the  machinegun  when  compared  to 
embedded  training  (A4)  ?  [ 1-9 , 1/2- 1/9  ] 


36.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  zero  the  machinegun  when  compared  to 
embedded  training  (A4) ?  C 1-9 , 1/2—1/93 


H.  ENGAGE  TARGETS  WITH  THE  COAX  FROM  THE  GUNNER'S 
STATION. 

STIMULI  PRESENTATION. 

I.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner's  station  when 
compared  to  part  task  computer  based  instruction  (A2) ? 
C 1-9 . 1/2-1/93 


2.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner’s  station  when 
compared  to  full  function  stand  alone  trainers  (A3) ? 
[1-9 , 1/2-1/9] 


3.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
present  information  required  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner’s  station  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/91 


4.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  coax  from  the  gunner’s  station  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/91 
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5.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  coax  from  the  gunner’s  station  when  compared  to 
embedded  training  (A4)?  C 1-9 , 1/2-1/9] 


6.  To  what  degree  or  level  of  potential  does  full 
function  stand  alone  trainers  (A3)  have  to  present 
information  required  to  train  the  task  engage  targets  with 
the  coax  from  the  gunner's  station  when  compared  to 
embedded  training  (A4)?  C 1-9 , 1/2-1/93 


RESPONSE  PRESENTATION 

7.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  coax  from  the  gunner’s 
station  when  compared  to  part  task  computer  based 
instruction  (A2) ?  C 1—9, 1/2-1/93 


8.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  coax  from  the  gunner's 
station  when  compared  to  full  function  stand  alone 
trainers  (A3)?  C 1—9, 1/2— 1/0] 


9.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  permit 
the  trainee  to  respond  to  information  presented  to  train 
the  task  engage  targets  with  the  main  gun  from  the 
gunner's  station  when  compared  to  embedded  training  (A4)'? 
[1-9 , 1/2-  1/9  ] 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
engage  targets  with  the  coax  from  the  gunner's  station 
when  compared  to  full  function  stand  alone  trainers  (A3) ^ 
[1-9 , 1/2—1/93 
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11.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  permit  the  trainee 
to  respond  to  information  presented  to  train  the  task 
engage  targets  with  the  coax  from  the  gunner's  station 
when  compared  to  embedded  training  (A4)?  [ 1-9 , 1/2- 1/9 ] 


12.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  permit  the  trainee  to 
respond  to  information  presented  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner’s  station  when 
compared  to  embedded  training  (A4)?  [1-9,1/2-1/91 


FEEDBACK  PRESENTATION 

13.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner's  station  when  compared  to 
part  task  computer  based  instruction  (A2)?  [ 1-9 , 1/2- 1/9  1 


14.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner’s  station  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [1-9,1/2-1/91 


15.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  feedback  required  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner's  station  when  compared  to 
embedded  training  (A4)?  [ 1-9 , 1/2-1/91 


10.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  coax 
from  the  gunner's  station  when  compared  to  full  function 
stand  alone  trainers  (A3)?  [1-9,1/2-1/91 


17.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  coax 
from  the  gunner’s  station  when  compared  to  embedded 
training  (A4) ?  [1-9,1/2-1/91 
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18.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  feedback 
required  to  train  the  task  engage  targets  with  the  coax 
from  the  gunner's  station  when  compared  to  embedded 
training  (A4)?  C 1-9, 1/2-1/93 


TASK  COMMONALITY. 

19.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  coax  from  the  gunner’s 
station  when  compared  to  part  task  computer  based 
instruction  (A2)?  [1-9,1/2-1/93 


20.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  coax  from  the  gunner’s 
station  when  compared  to  full  function  stand  alone 
trainers  (A3)?  [ 1-9 , 1/2- 1/9 ] 


21.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to  cover 
each  of  the  subtasks  (basic  steps)  required  to  train  the 
task  engage  targets  with  the  coax  from  the  gunner’s 
station  when  compared  to  embedded  training  (A4) ? 

C 1-9 . 1/2-1/9] 


22.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner’s  station  when 
compared  to  full  function  stand  alone  trainers  (A3)'’ 

C 1-9, 1/2— 1/9] 


23.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
tarJ-*«  «*<  th  the  coax  from  the  gunner's  station  when 
compared  to  embedded  training  (A4) ?  [1-9,1/2-1/93 


B  -  49 


24.  To  what  degree  or  level  of  potential  do  full  function 
atand  alone  trainers  (A3)  have  to  cover  each  of  the 
subtasks  (basic  steps)  required  to  train  the  task  engage 
targets  with  the  coax  from  the  gunner's  station  when 
compared  to  embedded  training  (A4)?  [ 1-9 , 1/2-  1/9  ] 


PHYSICAL  SIMILARITY. 

25.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necesstationry  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner's  station  when  compared  to 
part  task  computer  based  instruction  (A2) ?  [ 1-9 , 1/2- 1/9 3 


26.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necess tat i onry  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner's  station  when  compared  to 
full  function  stand  alone  trainers  (A3)?  [ 1 -9 , 1 / 2- 1 /9  ] 


27.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  have  to 
provide  physically  similar  displays,  controls,  or 
equipment  necesstationry  to  train  the  task  engage  targets 
with  the  coax  from  the  gunner’s  station  when  compared  to 
embedded  training  (A4) ?  C 1-9 , 1/2-1/93 


28.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necesstationry  to 
train  the  task  engage  targets  with  the  coax  from  the 
gunner's  station  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [ 1-9 , 1/2- 1/9  ] 


29.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necesstationry  to 
train  the  task  engage  targets  with  the  coax  from  the 
gunner's  station  when  compared  to  embedded  training  (A4)? 
C 1-9 , 1/2-1/9] 


30.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  have  to  provide  physically 
similar  displays,  controls,  or  equipment  necesstationry  to 
train  the  task  engage  targets  with  the  coax  from  the 
gunner's  station  when  compared  to  embedded  training  (A4) ? 
[1-9, 1/2- 1/9  ] 


TRAINING  TIME. 

31.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
coax  from  the  gunner's  station  when  compared  to  part  task 
computer  based  instruction  (A2)?  [ 1-9, 1/2— 1/9] 


32.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
coax  from  the  gunner's  station  when  compared  to  full 
function  stand  alone  trainers  (A3)?  [1-9,1/2-1/93 


33.  To  what  degree  or  level  of  potential  does  unit  or 
classroom  instructor  based  instruction  (Al)  require  less 
training  time  to  train  the  task  engage  targets  with  the 
coax  from  the  gunner’s  station  when  compared  to  embedded 
training  (A4)?  C 1-9 , 1/2- 1/03 


34.  To  what  degree  or  level  of  potential  does  part  task 
computer  baaed  instruction  (A2)  require  less  training  time 
to  train  the  task  engage  targets  with  the  coax  from  the 
gunner's  station  when  compared  to  full  function  stand 
alone  trainers  (A3)?  [ 1-9, 1/2-1/9D 


35.  To  what  degree  or  level  of  potential  does  part  task 
computer  based  instruction  (A2)  require  less  training  time 
to  train  the  task  engage  targets  with  the  coax  from  the 
gunner's  station  when  compared  to  embedded  training  (A4)7 
[1-9,1/2-1/93 


36.  To  what  degree  or  level  of  potential  do  full  function 
stand  alone  trainers  (A3)  require  less  training  time  to 
train  the  task  engage  targets  with  the  coax  from  the 
gunner’s  station  when  compared  to  embedded  training  (A4)? 
[1-9, 1/2-1/93 
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COMMENTS 


Please  Include  any  comments  in  the  space  below.  Comments 
might  include  the  ease  in  understanding  the  questionnaire, 
how  difficult  it  was  to  make  judgements,  any  ideas  on  an 
alternate  rating  scale  (perhaps  something  other  than 
pairwise  comparison),  alternate  rating  criteria,  and 
length  of  time  it  took  you  to  complete  the  questionnaire. 


APPENDIX  C.  TECHNIQUE  FOR  ORDER  PREFERENCE  BY  SIMILARITY 

TO  THE  IDEAL  SOLUTION  (TOPSIS) . 

This  appendix  presents  the  six  step  generalized 

formulation  of  the  multiattribute  decisionmaking  model 

TOPSIS  used  for  the  ordering  of  alternatives  when  cardinal 

preferences  for  the  criteria  (attributes)  are 
49 

available.  A  simple  example  of  the  application  of 
TOPSIS  is  also  presented. 

STEP  1.  Develop  a  normalized  decision  matrix.  This 

step  is  used  to  transform  the  various  measurement 

dimensions  of  the  criteria  (speed,  distance,  weight,  cost) 

into  nondimensional  (normalized)  units  so  that  comparisons 

across  criteria  can  be  made.  This  is  accomplished  by 

transforming  the  criterion  value  x  to  a  normalized 

i  J 

value  r  using  vector  normalization: 
ij 

r^*  =  _ Xjy. _ ,  V  J,  J  =  1,...,  n 

where  m  is  the 

number  of  alternatives 

and  n  is  the  number  of  criteria 

Step  2.  Construct  the  weighted  normalized  decision 

matrix.  The  normalized  decision  matrix  established  in 

Step  1  is  multiplied  by  a  set  of  decisionmaker  weights. 

These  weights  w=  (  w,  ,  w ...,wj,...,w7)  where  ^  w  =  1 

j  ••  i 

can  be  obtained  by  the  direct  choice  of  the  decisionmaker 


or  by  several  other  techniques  which  assist  the 

decisionmaker  in  establishing  these  weights.  Some 

examples  are:  the  eigenvector  method,  weighted  least 

square  method,  entropy  method  and  linear  programming 

techniques  for  multidimensional  analysis  of  preference 
50 

(LINMAP) .  The  normalized  decision  matrix  (R) 
multiplied  by  the  weights  (w)  becomes  the  normalized 
decision  matrix  (V) : 


f  W,  r.( 

W2r.Z 

1 

•••  Aj  •  •  • 
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where  m 

*  the 

number  alternatives 

and  n 

=  the 

number 

of 

the  criteria. 

Step  3.  Establish  the  ideal  and  negat ive- ideal 

solutions.  This  step  identifies  the  best  and  worst 

# 

solutions  by  forming  new  alternatives  (A  ,  A  ) 

comprised  of  the  highest  and  lowest  criteria  values 

associated  with  the  alternatives  under  study.  Remembering 

that  v  is  an  element  of  the  weighted  normalized  decision 

matrix  calculated  in  Step  2.  The  ideal  solution  is 

defined  as: 

« 

A  =  {max  vi/  ,  max  vu  ,  -  -  .  ,  max  v^„  }  , 

and  the  negative-ideal  solution  is  defined  as: 

A  =  {mitn  v^(  ,  min  v^  ^  ,  .  .  .  ,  mil)  v  'Ln  )  ,  where  i  = 


C 


2 


1,2 . m  alternatives  and  n 


1  ,  2  ,  .  .  .  , n  criteria . 


Note:  This  definition  is  for  benefit  criteria.  Cost 

criteria  take  the  opposite  solution  definition. 

Step  4.  Calculate  the  separation  measure.  This  step 
calculates  the  distance  between  each  alternative  and  both 
the  ideal  and  negative- ideal  solutions.  It  is  an 
Euclidean  distance  in  n  (number  of  criteria)  dimensions. 
From  the  ideal  solution  this  separation  measure  is: 


from  the  negative- ideal  solution  the  separation  measure 


Step  5.  Determine  the  relative  closeness.  This 
calculation  establishes  the  c  value  or  the  relative 
closeness  of  a  particular  alternative  to  the  ideal  one. 

It  is  given  by: 

c^4  =  s^_  /  ( s +  s^_  ),  o  <  <  1  ,  i  =  1,2,.  .  .  ,  m 

If  there  are  two  alternatives  with  the  same  separation 
measure  to  the  ideal  solution  (s;4l)  then  the  alternative 

v 

that  is  also  the  farthest  from  the  negative- ideal  solution 
(i.e.  has  the  larger  separation  measure,  s^_  wi 1 1  be 
selected . 

Step  6.  Rank  the  preference  order.  The  alternatives 
are  ranked  first  to  last  in  the  order  of  the  largest  to 
the  smallest  c  value. 
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A  short  example  will  demonstrate  the  calculations 


involved  in  TOPSIS.  Let  us  say  we  are  trying  to  select  a 
car  among  three  alternatives  (  A1 ,  A2 ,  A3)  and  we  decide 


that 

price , 

fuel  economy,  and 

top  speed  (xl,x2,x3)  will  be 

the 

criteria 

we  use  to 

j  udge 

the  alternatives.  Our  sample 

data 

i  s  : 

xl 

x2 

x3 

A 1 

r  30 

15 

125  : 

D 

=  A2 

!  20 

24 

i io  ; 

A3 

L  10 

29 

95  J 

STEP  1 

Develop  a  normalized 

decision  matrix. 

xl 

x2 

x3 

A 1 

r  .  802 

.  370 

.  652  *3 

R 

=  A2 

:  .  534 

.  592 

.  574  ; 

A3 

^  .  267 

.716 

.496  J. 

where 

p//  = 

-JLu— 

30 

30  =  .802 

1 

f  2 

A  x4/ 

t  :l 

o 

n 

+  (20)2  +  (  lof 

1400 

STEP  2.  Construct  the  weighted  normalized  decision 


matrix.  For  our  example,  we  assume  that  the  relative 
importance  of  the  criteria  is  w  =  (  .4,  .3,  .3) . 

Therefore,  V  becomes: 


x 1  x2  x3 


A 1 

.  802 

(  .  4) 

.370  (.3) 

652  ( . 3 )  : 

V  = 

A2 

.  534 

(  .  4) 

.592  (.3) 

574  (.3)  : 

A3 

.  267 

(  .  4) 

.716  ( . 3) 

496  (.3)  1 

xl 

x2 

x3 

A 1 

.  32  1 

.111 

.  196  : 

V  = 

A2 

.2  14 

.  178 

.  172  ! 

A3 

.  107 

.2  15 

.  149  : 

STEP  3. 

Es tab  1 i sh  the 

ideal 

and  negative 

-ideal  solutions 

C 
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The  ideal  solution  contains  the  largest  v  in  each  column 

(except  v  which  is  a  cost  criterion  and  takes  the 
i  1  * 

smallest  value):  A  =  (  .107,  .215,  .196). 

The  negative  ideal  solution  contains  the  smallest  v  in 

each  column  (except  v  which  is  a  cost  criterion  and 

i  1 


takes  the  largest  value):  A  =  (  .321,  .111,  .149). 

STEP  4.  Calculate  the  separation  measure. 


i 

t 

r  * 

(v,  -  V. 

J 

)2 

j 

S.A  = 

[(.321  -  .  107)  + 

(  . 

1  1  1 

-  .  215)  + 

( .  196  -  , 

.  196) 

] 

= 

.  238 

SZ*  = 

[ ( . 214  - .  107)  + 

(  . 

178 

-.215)  + 

( . 172  - , 

.  196) 

] 

= 

.  116 

S3*  = 

[ ( . 107  - . 107)  + 

(  . 

215 

-.215)  + 

( . 149  - , 

.  196) 

1 

J 

= 

.  047 

and 

u 

Z 

S  ' 

jC  ~ 

1 

<vj  -  vf 

) 

s,'  = 

[  (  .321  -.321)  + 

(  . 

1 1 1 

-.111)  + 

( .  196  -  . 

.  149) 

1 

= 

.051 

S2'  = 

[  (  . 214  -  .321)  + 

(  . 

178 

-.111)  + 

( . 172  - . 

.  149) 

] 

= 

.  128 

s,-  = 

[  ( . 107  -  .321)  + 

(  . 

215 

-.111)  + 

( .  149  - . 

.  149) 

] 

s 

.  238 

STEP 

5 .  Determine  the 

relative  closeness. 

ci*  = 

/  +  s,  -  ) 

= 

• 

051/ ( . 238 

+  .051) 

= 

176 

C2*  = 

. 128/ (.116  +  .128) 

= 

.  525 

c.^  = 

. 238/ ( . 047  +  .238) 

= 

.  835 

STEP  6.  Rank  the  preference  order. 

The  alternatives  are  ranked  from  step  5: 


First: 

A3 

( . 835) 

Second: 

A2 

( . 525) 

Third : 

A 1 

( . 176) 
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APPENDIX  D:  QUESTIONNAIRE  DATA. 

This  appendix  provides  the  questionnaire  input  data 
and  eigenvector  calculations  for  the  level  3  (  alternative 
versus  criteria)  weight  determinations.  Each  page 
provides  the  six  pairwise  comparisons  made  for  each 
subject  matter  expert  for  each  task. 

The  input  data  is  shown  in  matrix  format.  For 
example,  the  first  matrix  on  page  D  -  3  reflects  the 
following  SME  input  for  the  stimuli  presentation  input  for 


the  first  task: 

Instructor  Based  Instr  :  Computer  Based  Instr  =  1/4 

Instructor  Based  Instr  :  Stand  Alone  Trainers  =  1/7 

Instructor  Based  Instr  :  Embedded  Training  =  1/9 

Computer  Based  Instr  :  Stand  Alone  Trainers  =  1/2 

Computer  Based  Instr  :  Embedded  Training  =  1/5 

Stand  Alone  Trainers  :  Embedded  Training  =  1 

The  entry  for  c  provides  the  consistency  index.  For  the 


matrix  1ASP  (the  first  one  next  page)  the  consistency 
index  is  .037. 


D 
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LEGEND 


1ASP  means: 

1  =  SME  #  1 
A  =  Task  A 

SP  =  Stimulus  Presentation 
FP=  Feedback  Presentation  SP= 

PS=  Physical  Similarity  TC= 

RP=  Response  Presentation  TT= 

CBI=  Computer  Based  Instruction 
IB=  Instructor  Based  Instruction 
FFSAT=  Stand  Alone  Trainer 


3CTT  means : 

3  =  SME  *  3 
C  =  Task  C 
TT  =  Training  Time 
Stimulus  Presentation 
Task  Commonality 
Training  Time 
ET=  Embedded  Training 


DATA  CONTENTS: 


Sub j  ect 
Task 
Sub j  ect 
Task 
Sub j  ect 
Task 
Subject 
Task 
Subject 
Task 
Subj  ect 
Task 


Matter  Expert  #  1: 

A  -  H . 

Matter  Expert  *  2: 

A  -  H . 

Matter  Expert  #  3: 

A  -  H . 

Matter  Expert  *  4: 

A  -  H . 

Matter  Expert  #  5: 

A  -  H . 

Matter  Expert  *  6: 
A  -  H . 


D  -  3  through  D  -  10 


D  -11  through  D  -  18 


D  -19  through  D  -  26 


D  -27  through  D  -  34 


D  -35  through  D  -  42 


D  -43  through  D  -  50 


D 
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The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

ET 

IB 

I  .  00 

0 . 25 

0.14 

0.11 

CBI 

4 . 00 

1 . 00 

0 . 50 

0 . 20 

FFSAT 

7 . 00 

2.00 

1  .  00 

1  .  00 

ET 

9 . 00 

5 . 00 

1 . 00 

1  .  00 

The  DM  subjective  weights 
r  B  CBT  FFSAT  ET 


0 .05 

0.15 

0 . 35 

0 . 46 

C  =  ,03 

The  Eigenvector 
IB 

Pa l rwi s  e 
CBI 

compar isons 
FFSAT  ET 

IB 

1  .  OC 

0 . 20 

0.14 

0.14 

CBI 

5 . 00 

1 . 00 

0 . 25 

0 . 25 

FFSAT 

7 . 00 

4 . 00 

1  .  00 

1 . 00 

ET 

7 . 00 

4 . 00 

1 . 00 

1 . 00 

The  DM  subjective  weights 

IB  CBI  FFSAT  ET  ^  ^  o  -  4 

0.05  0.14  0.41  0.41 


The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

ET 

IB 

1  .  00 

0 . 20 

0.14 

0.14 

CBI 

5 . 00 

1 .00 

0 . 25 

0 . 25 

FFSAT 

7 . 00 

4 . 00 

1 . 00 

1 . 00 

ET 

7 . 00 

4 . 00 

1  .  00 

1  .  00 

The  DM 

subj  ecti ve 

weights 

i  AFP 

IB 

CBI 

FFSAT  ET 

0 . 05 

0.14 

0.41 

0.41 

C  -  ■  05  4 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1 . 00 

5 . 00 

3 . 00 

3.00 

CBI 

0 . 20 

1 . 00 

0 . 33 

0.33 

FFSAT 

0.33 

3.00 

1  .  00 

1  .  00 

ET 

0.33 

3 . 00 

1  .  00 

1  .  00 

The  DM 

subjective 

weights 

IB 

CBI 

FFSAT  ET 

/  070 

0 . 52 

0 . 08 

0 . 20 

0 . 20 

c-.oiy 

The  Eigenvector 

p, 

airwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1 . 00 

0 . 20 

0.11 

0.11 

CBI 

5 . 00 

1  .  00 

0 . 20 

0 . 20 

FFSAT 

9 . 00 

5 . 00 

1  .  00 

1 . 00 

ET 

9 . 00 

5 . 00 

1  .  00 

1  .  00 

The  DM 
13 

subjective  weights 
CBI  FFSAT  ET 

/  APS 

o 

o 

0.12 

0 . 42 

0 . 42 

,0  57 

The  Eigenvector 

Pairwise  compar i: 

3  o  ns 

I  B 

CBI 

FFSAT 

ET 

13 

1  .  00 

0 . 33 

0 . 20 

0  .  20 

co: 

3  CO 

1  .  00 

0 . 33 

0  33 

FFSAT 

5 . 00 

3 . 00 

1 . 00 

:  .  oc 

ET 

5 . 00 

3 . 00 

1 . 00 

i  .  00 

The  DM 

sub j  ec 1 1  v 

’  e 

weights 

13 

CBI 

FFSAT  ET 

/  qrr 

0 . 07 

C.  15 

0.39 

0 . 39 

C  -  -  0s  b 

D  -  3 


The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

ET 

IB  I. 00 

0 . 25 

0.14 

0.11 

CBI  4.00 

1 . 00 

0 . 50 

0 . 20 

FFSAT  7.00 

2 . 00 

1 . 00 

1 . 00 

ET  9.00 

5 . 00 

1 . 00 

1 . 00 

The  DM  subjective 

weights 

IAS  f 

IB  CBI 

FFSAT  ET 

0.05  0.15 

0 . 35 

0 . 46 

C- i03> 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB  1.00 

0 . 20 

0.14 

0.14 

CBI  5.00 

1  .  00 

0 . 25 

0 . 25 

FFSAT  7.00 

4 . 00 

1  .  00 

1  .  00 

ET  7.00 

4 . 00 

1  .  00 

1 . 00 

The  DM  subjective 

weights 

1 

IB  CBI 

FFSAT  ET 

0.05  0.14 

0.41 

0.41 

M  -OS1/ 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB  1.00 

0 . 20 

0.14 

0.14 

CBI  5.00 

1 . 00 

0 . 25 

0 . 25 

FFSAT  7.00 

4 . 00 

1  .  00 

!  .  00 

ET  7.00 

4 . 00 

1  .  00 

: .  oo 

The  DM  subjective 

weights 

1 

IB  CBI 

FFSAT  ET 

0.05  0.14 

0.41 

0.41 

C- .OSH 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB  1.00 

5.00 

3.00 

3.00 

CBI  0.20 

1 .00 

0 . 33 

0 . 33 

FFSAT  0.33 

3 , 00 

1  .  00 

1  .  00 

ET  0.33 

3.00 

1  .00 

1 . 00 

The  DM  subjective 

weights 

IB  CBI 

FFSAT  ET 

/  LtfC 

0.52  0.08 

0 . 20 

0 . 20 

C'.OlIc 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB  1.00 

0 . 20 

0.11 

0.11 

CBI  5.00 

1 . 00 

0 . 20 

0 . 20 

FFSAT  9.00 

5 . 00 

1 . 00 

1  .  00 

ET  9.00 

5 . 00 

1 . 00 

1  .  00 

The  DM  subjective 

weights 

IB  CBI 

FFSAT  ET 

/  h  f  'Z 

0.04  0.12 

0 . 42 

0.42 

c  -  ■  0*>  / 

The  Eigenvector 

Pairwise  compar isoj  3 

13 

CBI 

FFSAT 

ET 

IB  1.00 

0 . 33 

0 . 20 

0 . 20 

CBI  3.00 

1 . 00 

0 . 33 

0 . 33 

FFSAT  5.00 

3 . 00 

1 . 00 

1  .  00 

ET  5.00 

3 . 00 

1  .  00 

1  .  00 

The  DM  subjective 

weights 

nYn 

IB  CBI 

FFSAT  ET 

0.07  0.15 

0 . 39 

0 . 39 

c-.Oii 

D  -  4 


The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

ET 

IB 

1 . 00 

1 . 00 

0 . 33 

0 . 33 

CBI 

1 . 00 

1 . 00 

0 . 50 

0 . 50 

FFSAT 

3 . 00 

2 . 00 

1  .  00 

1  .  00 

ET 

3.00 

2 . 00 

1 . 00 

1  .  00 

The  DM  subjective  weights  „ 

IB  CBI  FFSAT  ET  < 

0.13  0.1S  0.35  0.35  C  .  0Ot 


The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

ET 

IB 

1 . 00 

0 . 50 

0 . 33 

0 . 33 

CBI 

2 . 00 

1 . 00 

0 . 50 

0 . 50 

FFSAT 

3 . 00 

2 . 00 

1  .  00 

0 . 50 

ET 

3 . 00 

2 . 00 

2 . 00 

1  .  00 

The  DM 

aubj  ecti ve 

weights 

ICRf 

IB 

CBI 

FFSAT  ET 

0.11 

0.19 

0 . 29 

0.41 

c--.ou , 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1.00 

0 . 20 

0.14 

0.14 

CBI 

5 . 00 

1 . 00 

0 . 33 

0.33 

FFSAT 

7 . 00 

3 . 00 

I  .  00 

1 . 00 

ET 

7.00 

3 . 00 

1  .00 

1  .  00 

The  DM 

subjective 

weights 

>  Crr 

IB 

CBI 

FFSAT  ET 

0.05 

0.  16 

0 . 39 

0 . 39 

C  :.0C4 > 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1 . 00 

3.00 

1  .  00 

1 . 00 

CBI 

0 . 33 

1 . 00 

0 . 50 

0.50 

FFSAT 

1 .00 

2.00 

1 . 00 

1 . 00 

ET 

1 . 00 

2 . 00 

1  .  00 

1  .  00 

The  DM 

subj  ecti ve 

weights 

/  076 

IB 

CBI 

FFSAT  ET 

0.31 

0.  13 

0 . 28 

0 . 28 

c-.OOf? 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

I  .  00 

1 . 00 

0 . 33 

0 . 33 

CBI 

1 . 00 

1  .  00 

0 . 50 

0 . 50 

FFSAT 

3 .00 

2.00 

1  .  00 

1  .  00 

ET 

3 . 00 

2 . 00 

1  .  00 

1  .  00 

The  DM 

subjective 

weights 

it?' 

IB 

CBI 

FFSAT  ET 

0.13 

0.16 

0 , 35 

0 . 35 

■ r-.co 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1  .  00 

0 . 50 

0  25 

0.25 

CBI 

2 . 00 

1 . 00 

0 . 50 

0.50 

FFSAT 

4 . 00 

2 . 00 

1 .00 

:  .oo 

ET 

4 . 00 

2 . 00 

1  .  00 

l  .  00 

The  DM 

subjective 

weights 

/  crT 

IB 

CBI 

FFSAT  ET 

0 . 09 

0.18 

0 . 36 

0 . 36 

c  -  o 

D  -  5 


The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT  ET 

IB 

1  00 

1 . 00 

1.00  1.00 

C8I 

1 . 00 

1 . 00 

1.00  1.00 

FFSAT 

: .  oo 

I  .00 

1.00  1.00 

ET 

1 . 00 

I  .  00 

1.00  1.00 

The  DM 

subjective  weights 

1  0  D  f 

IB 

CBI 

FFSAT  ET 

0 . 25 

0 . 25 

0 . 25 

0.25  C-C 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT  ET 

IB 

1 . 00 

0.33 

0.14  0.20 

CBI 

3 . 00 

1 . 00 

0.33  0.50 

FFSAT 

7 . 00 

3 . 00 

1.00  2.00 

ET 

5 . 00 

2 . 00 

0.50  1.00 

The  DM 

s ub j  ec  1 1 v 

e  weights 

>0 

IB 

CBI 

FFSAT  ET 

0 . 06 

0.16 

0 . 49 

0.29  (.  r.C''> 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT  ET 

IB 

1  .  00 

0 . 33 

0.14  0.20 

CBI 

3 . 00 

I  .  00 

0.33  0.50 

FFSAT 

7 . 00 

3  00 

1.00  2.00 

ET 

e  ••• 

.1 . 00 

0.50  1.00 

The  D.V. 

t:  c  '  e  c  t ;  v 

e  we  :  g  h  t  s 

i- 

IB 

CBI 

FFSAT  ST 

!  0  rf" 

0 . 06 

0.15 

0. 49 

0.29  :-,£0'v 

The  Eigenveri^- 

Pairwise  comparisons 

13 

CHI 

FFSAT  ET 

IB 

1  .  00 

1 . 00 

0.50  1.00 

cb  r 

1  .  oo 

1 . 00 

0.50  1.00 

FFSAT 

2 . 00 

2 . 00 

1.00  2.00 

ET 

1  .  00 

;  oo 

0.50  1.00 

The  DM 
IB 

subjective  weights 
C3I  FFSAT  ET 

}  0  rc 

0 . 20 

0 . 20 

0 . 40 

o  .20  •;  »  O 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT  ET 

IB 

1  .  00 

v  .  25 

0.14  0.14 

CBI 

4 . 00 

1  .  9C 

0.5C  0.50 

FFSAT 

7 . 00 

2 . 00 

1.00  .00 

ET 

7 .00 

2 . 00 

1.C0  1.00 

The  DM 

subjective  weights 

-  S 

r*  g  T 

PFS AT  ET 

!  r  j 

0  0P 

0  .  1  9 

'  38 

to  .  .  y 

The  E..1 

e  r.  v  e  c  t  o  r 

Pairwise  c 

:  a 

: 

Tr CAT  E? 

IB 

:  oo 

?  -  0 

~x  "  n 't 

cb: 

■: .  o  o 

:  oo 

: .  t  *.  o .  5  o 

FFSAT 

3  .  oo 

.. .  oo 

;  :  •  ••  c 

E~' 

•  .00 

3  .  JO 

The  DM 

subjectiv 

e  weights 

IB 

CBI 

FFSAT'  ET 

f  0 

0.11 

0.19 

0  35 

0.35  t_z  sXy'  •- 
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The  Eigenvector 

?a 

i rwi se 

compar i s 

o  ns 

IS 

CBI 

FFSAT 

ET 

:e 

1  .  00 

0 . 25 

0.13 

0.13 

CBI 

4 . 00 

1 . 00 

0 . 50 

0 . 50 

FFSAT 

3 . 00 

2 . 00 

1  .  00 

1  .  00 

ET 

8  .  CO 

2 . 00 

1 . 00 

1.00 

The  DM 

subjective 

we  l  g  h  t 

s 

--  .  r 

13 

CBI 

FFSAT 

ET  / 

■“  o  * 

0 . 05 

0.19 

0 . 38 

0 . 33 

(  ~o 

The  Eigenvector 

Pairwise 

comparisons 

IB 

CBI 

FFSAT 

ET 

13 

1  .  00 

0 . 25 

0.13 

0.13 

CBI 

4 . 00 

1  .  00 

0 . 50 

0 . 50 

FFSAT 

8 . 00 

2 . 00 

1 . 00 

I  .  00 

ET 

8 . 00 

2 . 00 

1 . 00 

1 . 00 

The  DM 

subjective 

weights 

13 

CBI 

FFSAT 

ET  / 

F  r.  f 

0 . 05 

0  .  19 

0 . 38 

0 . 38 

c~c 

The  Eigenvector 

Pai rwi se 

comparisons 

IB 

CBI 

FFSAT 

ET 

IB 

1  .  00 

0. 25 

0.13 

0.13 

CBI 

4 . 00 

I  .  00 

0 . 50 

0 . 50 

FFSAT 

8 . 00 

2 . 00 

1  .  00 

1  .  00 

ZT 

8  .  '  0 

2 . 00 

•  r 

1  0 .' 

The  3  M 

sub  '  6  r  *.  i 1 

v  e 

we i £  h t 

3 

—  r 

IB  CBI 

ffsat" 

ET  f  7 

0.05  0.19 

0.38 

0 . 38 

The  Eigenvector 

Pa l rwi se 

compar isons 

13 

CBI 

FFSAT 

ET 

IB  .  00 

0 . 33 

0 . 20 

0 . 20 

CBI  3.00 

1  .  00 

0 . 50 

0  50 

FFSAT  5.00 

2 . 00 

i  .  00 

1  .  00 

ET  5 . 00 

2.00 

1  .  00 

: .  oo 

the  DM  subjective  weights 


IB  CBI 

ffsat"  ET 

!  ?■(. 

0.07  0.19 

0  .  3~ 

:•  .37  _ 

.  2<i  ?. 

The  Eigenvector 

Pairwise  r 

ompar i sons 

I  S 

CBI 

FFSAT 

ET 

13  1,00 

0  .  33 

0 . 20 

9  .  20 

CBI  3.00 

:  .00 

9.50 

0 . 50 

FFSAT  5.00 

2 . 00 

1  .  00 

: .  oc 

ET  5.00 

2 . 00 

; .  oo 

1  .  00 

The  DM  subject; 

v  e  we  i  g  h  t  s 

.  -  P 

13  CBI 

FFSAT  ET 

■;  .  ">  7  0  1 .. 

■  ■  v;  ,  « 

,  a 

The  Bi2envecttr 

Fair  w :  .v  e  : 

,B5;,r  i tr 

•.  r 

-  -2 

J.  ■ 

—  r."  ?  * 

-  -  * 

•  •  —  . 

•-  * 

:  -  :  •  - 

0  ;  ? 

0 

0 

j  •»  z,  *  1  A 

.  -*  V 

.  VO 

FFS: AT  5  -  > 

2  vO 

'  '  n 

1  V 

ET  5  X* 

2  .  >c 

:  CO 

The  TV  subject: 

v  e  we  .  £  r.  t  s 

>  -  rl 

13  rB : 

FFSAT  ~  4 

< j  ~  •;  ;  y 

,  .  2'-’ 

0.27  £ 

r  .  L  C  .' 
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The  Eigenvector  Pairwise  comparisons 


IB 

CBI 

FFSAT 

C.T 

IB 

1 . 00 

1  .  00 

1  .  00 

I  .  00 

CBI 

1 . 00 

1 . 00 

: .  oo 

I  .  00 

FFSAT 

1 . 00 

1  .  00 

1  .  00 

I  .  00 

ET 

1  .  00 

1  .  00 

1  .  00 

1  .  oc 

The  DM  subjective  weights 
IB  CBI  FFSAT  ET 

0.25  0.25  0.25  0.25 


The  Eigenvector 

p 

airwise  c 

ompar 1 sons 

IB 

CBI 

FFSAT  ET 

IB 

1  .  00 

0 . 50 

0.50  0.50 

CBI 

2 . 00 

1 . 00 

1.00  1.00 

FFSAT 

2 . 00 

1 . 00 

1.00  I  . 00 

ET 

2 . 00 

1 . 00 

1  .  C  0  1.00 

The  DM 

sub j  ec 1 1 ve 

weights 

/  6  :■? 

IB 

CBI 

FFSAT  ET 

0.14 

0 . 29 

0 . 29 

0.29  r-'-C 

The  Eigenvector 

Pairwise  comparisons 

IB 

CBI 

FFSAT  ET 

IB 

1 . 00 

0  .  50 

0.50  0.50 

CBI 

2.00 

1  .  00 

1.00  1.00 

FFSAT 

2 . 00 

: .  00 

:.oc  i.oo 

ET 

- 

;  o  >; 

The  DM 

sub; 

e c t  i 

we  ;  2  h  t  s 

:r  <-  f 

IB 

CBI 

FFSAT  ET 

0.14 

0 . 29 

0 . 29 

0.29 

The  Eigenvector 

P 

airwise  comparisons 

13 

CBI 

FFSAT  ET 

13 

.  00 

0 . 50 

0.50  0.50 
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APPENDIX  E:  TOPSIS  INDIVIDUAL  RANKINGS 

This  appendix  provides  the  individual  subject  matter 
expert  rankings  of  the  training  alternatives  for  each  task 
using  the  Technique  for  Order  Preference  by  Similarity  to 
the  Ideal  Solution. 

The  values  in  the  decision  matrix  were  obtained  from 
the  DM  subjective  weights  found  in  Appendix  D. 

The  Eigenvector  Pairwise  comparisons  section  provides 
the  criteria  versus  goal  pairwise  comparison  input  from 
that  portion  of  the  questionnaire.  It  is  the  same  for 
each  subject  matter  expert. 

The  DM  subjective  weights  in  this  appendix  are  the 
results  of  the  eigenvector  calculation  of  the  criteria 
importance  and  are  the  weights  used  in  Step  2  of  TOPSIS 
(see  Appendix  C) . 

The  rank  order  section  provides  the  subject  matter 
experts  alternative  ranking  for  that  particular  task  based 
on  c  values  or  relative  closeness  to  the  ideal  solution. 
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Respons 

Feedbac 

Task  Co 

Phy  Sim 

Tng  T  i 

0*.  04 

0 . 05 

0 . 04 

0.04 

0 . 04 

0.  14 

0.13 

0.14 

0.14 

0.14 

0 . 32 

0 . 34 

0  .  32 

0 . 35 

0 . 35 

0 . 50 

0 . 48 

0 . 50 

0 . 46 

0 . 45 

Co  Phy  Sim  Tng  Tim 
1.00  9.00 

3.00  9.00  , 

i  .  00  9.00  £r  .  I*1  * 

1.00  1.00 

1.00  1.00 

1.00  1.00 


Tng  Tim 
0 . 05 


The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.57  0.50  0.48  0.50  0.46  0.46 


The  Negative- Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.04  0.04  0.05  0.04  0.04  0.04 


Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Stand  Alone  Trnr  0.58 

Comp  Based  Instr  0.19 

Instr  Based  Instr  0.00 

^  /n  C.  $  .  *> 

TA3K  8 
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The  Decision  Matrix 


Instr  Based  :nstr 
Comp  3ased  Instr 
Stand  Alone  Trnr 
Embedded  Tng 


Stimuli  Respons 
0.50  0.50 

0.25  0.25 

0.19  0.06 

0.06  0.19 


Feedbac  Task  Co 
0.44  0.38 

0.28  0.43 

0.05  0.10 

0.23  0.09 


Phy  Sim 
0.49 
0 . 23 
C  .  06 
0 . 22 


The  Eigenvector  Pairwise  comparisons 


S  t i mul i 
Respons 
Feedbac 
Task  Co 
Phy  Sim 
Tng  Tim 


Stimuli 

Respons 

Feedbac 

Task  Co 

Phy  Sim 

1 . 00 

1 . 00 

0 . 33 

0 . 33 

1 . 00 

1  -  00 

1 .00 

1 . 00 

0 . 33 

1  .  00 

3 . 00 

1 . 00 

1 .00 

0 . 33 

1  .  00 

3.00 

3 . 00 

3 . 00 

1 . 00 

1 . 00 

1.00 

1 . 00 

1 . 00 

1 . 00 

1 .00 

1 . 00 

3 . 00 

1  .  00 

3 . 00 

1 . 00 

Tng  Tim 
1  .  00 
0 . 33 

)  00  C-JH, 

0 . 33 
1 . 00 
1 . 00 


The  DM  subjective  weights 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0 ■  I  1  0.11  0.16  0.24  0.15  0.24 


The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  phy  Sim  Tng  Tim 

0.50  0.50  0.44  0.43  0.49  0.42 


The  Negative- Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.06  0.06  0.05  0.09  C  .  06  0.08 


Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Instr  Based  Instr  0.93 

Comp  Based  Instr  0.67 

Embedded  Tng  0.23 

Stand  Alone  Trnr  0.09 


T.-.g  Tiro 
0.42 
0  .  42 
0 . 08 
0 . 08 
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The  Decision  Matrix 


3 

1 1  mu  1  i 

Respons 

Feedbac  T 

ask  Cc 

Phy  Sim 

Tng  Ti: 

Instr 

Based 

Instr 

0 . 05 

0 . 05 

0 . 05 

0  .  13 

0 . 07 

0 . 08 

Comp 

Based 

Instr 

0 . 57 

0.61 

0 . 24 

0  .  27 

0.61 

0 . 54 

Stand  Alone 

Trnr 

0.14 

0  .  13 

0.14 

0 . 07 

0.12 

0.14 

Embedded  Tng 

0 . 24 

0.21 

0 . 58 

0 . 53 

0 . 20 

0 . 24 

The  Eigenvector  Pairwise 

corapari sons 

Stimuli 

Respons  Feedbac 

Task 

Co  Phy  Sim  Tng  Tim 

S 1 1 mu 1 i 

1 . 00 

1  .  00 

5 . 00 

4 . 00 

3.00 

5 . 00 

Respons 

1 . 00 

1 . 00 

0 . 25 

0 . 25 

0 . 50 

0 . 25 

Feedbac 

0 . 20 

4 . 00 

1 . 00 

5 . 00 

5 . 00 

5 . 00 

Task  Co 

0.25 

4 . 00 

0 . 20 

1 . 00 

1 . 00 

3 . 00 

Phy  Sim 

0 . 33 

2 . 00 

0 . 20 

1 . 00 

1 . 00 

5 . 00 

Tng  Tim 

0.20 

4 . 00 

0.20 

0 . 33 

0 . 20 

1  .  00 

The  DM  sub>ective  weights 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.34  0.09  0.27  0.12  0.12  0.07 


The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.57  0.61  0.58  0.53  0.61  0.54 

The  Negat ive- Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.05  0.05  0.05  0.07  0.07  0.08 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Comp  Based  Instr  068 

Embedded  Tng  0.57 

Stand  Alone  Trnr  0.16 

Instr  Based  Instr  0.03 

S/r~i  £  y  i 
Tfi-s*  8 
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Tn  i  Tirr. 
0-09 
0 . 20 
0  .  35 
0 . 35 

The  Eigenvector  Pairwise  comparisons 


Stimuli  Respons  Feedbac  Task 

Co  Phy  Sim 

Tng  T 

Stimuli 

1.00  1.00  0.20 

0 . 33 

0 . 33 

0 . 33 

Reapons 

1.00  1.00  0.20 

0.33 

1  .  00 

0 . 33 

Feedbac 

5.00  5.00  1.00 

3 . 00 

3 . 00 

3 . 00 

Taak  Co 

3.00  3.00  0.33 

1 . 00 

1  .  00 

1  .  00 

Phy  Sim 

3.00  1.00  0.33 

1 . 00 

1 . 00 

1  .  00 

Tng  Tim 

3.00  3.00  0.33 

1 . 00 

1 . 00 

1 . 00 

The  DM  subjective  weights 

Stimuli 

Reapons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0 . 06 

0.08  0.40  0.16 

0  .  14 

0 . 16 

The  Ideal  Solution 

Stimuli 

Reapons  Feedbac  Taak  Co 

Phy  Sim 

Tng  Tim 

0.56 

0.49  0.43  0.40 

0.56 

0 . 35 

The  Negat ive- Ideal  Solution 

Stimuli 

Raapona  Faadbac  Taak  Co 

Phy  Sim 

Tng  Tim 

0 . 06 

0.07  0.07  0.09 

0.06 

0 . 09 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Stand  Alone  Trnr  0.74 

Comp  Baaed  Inatr  0.18 

Inatr  Baaed  Inetr  0.03 


S 1 1 mu  1 1 

Respons 

F  eedbac 

Task  Co 

Phy  Sim 

Instr  Based  Instr 

0.12 

0.07 

0 . 07 

0  . :  2 

0 . 06 

Comp  Baaed  Instr 

0 . 06 

0.15 

0  .  13 

0 . 09 

0  12 

Stand  Alone  Trnr 

0 . 26 

0 . 29 

0 . 37 

0 . 40 

0 . 26 

Embedded  Tng 

0 . 56 

0 . 49 

0.43 

0 . 40 

j  56 
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task  & 
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St ; mui 2 

Respons 

F eedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

I  n  s  t  r 

Based 

I  n  s  t  r 

0  13 

0.11 

0 . 05 

0.31 

0  13 

0  -  09 

Comp 

3ased 

Ins  t  r 

0  .  16 

0.19 

0.16 

0  .  13 

0.16 

0.18 

Stand  Alone 

Trnr 

0 . 35 

0 . 29 

0 . 39 

0 . 28 

0 . 35 

C  .  36 

Embedded  Tng 

0 . 35 

0.41 

0 . 39 

0 . 28 

0 . 35 

C  ■  36 

The  Eigenvector  Pairwise 

compar l sons 

Stimuli 

Respon 

s  Feedbac  Task 

Co  Phy  ? 

:  m  Tng  T 

i  m 

S 1 1 mu 1 i 

I  .00 

1 . 00 

1 . 00 

0  33 

3 . 00 

4 . 00 

Re  spons 

1 . 00 

1  .  00 

1  .  00 

0.33 

3 . 00 

4 . 00 

F eedbac 

1 . 00 

1 . 00 

: .  oo 

0 . 33 

3 . 00 

4  00 

C-  .Oil 

Task  Co 

3.00 

3 . 00 

3 . 00 

1  .  00 

5 . 00 

5. 00 

Phy  Sim 

0.33 

0 . 33 

0 . 33 

0.20 

1 . 00 

3 . 00 

Tng  Tim 

0.25 

0 . 25 

0 . 25 

0 . 20 

0 . 33 

1  .  00 

The  DM  subjective 

weights 

S timul i 

Respong  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0  .  16 

0  .  16 

0.  16 

0 . 39 

0  .  O'? 

0 . 04 

The  Ideal  Solution 

Stimuli  Reapona  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.35  0.41  0.39  0.31  0.35  0.36 


The  Negat ive- Ideal  Solution 

Stimuli  Reapona  Feedbac  Taak  Co  Phy  Sim  Tng  Tim 
0.13  0.11  0.05  0.13  0.13  0.09 

Rank  Order:  Relative  Cloaeneas  to  the  Ideal  Solution 


Embedded  Tng  0.89 

Stand  Alone  Trnr  0.81 

Inatr  Baaed  Inatr  0.47 

Comp  Baaed  Inatr  0.19 
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Matrix 


The  Decisi'i 


Stimuli  Respons  Feedbac  '.ask  To  ?hy  Sim  Tr.g  "'im 


.s  tr 

Baaed  Instr 

0 . 04 

0 . 03 

0  .  29 

v'  .  0  4 

0 . 54 

0  .  bd 

Comp 

3ased  Instr 

0.12 

0.13 

0.06 

o .  i : 

0 . 33 

9 . 08 

Stand 

Alor.e  Trnr 

0  3  1 

0.90 

0.20 

0.49 

0  .  OS 

0.15 

Embedded  Tng 

0 . 53 

0 . 55 

0.44 

C  .  36 

0  .  Of 

0  .  19 

The  Eigenvector  Pairwise  comparisons 

Stimuli  Hespons  Feedbac  Task  Co  Phy  Sim  Tr»g  T :  .r. 


Stimuli 

1.00  : . 00 

0.11 

0.14 

0  .  14 

0  2  0 

Respons 

1.00  1.00 

0 . 33 

0 . 20 

0 . 20 

0  33 

Feedbac 

9.00  3.00 

: .  oo 

0 . 33 

0  33 

0.25 

Task  Co 

7.00  5.00 

3 . 00 

1  00 

3 . 00 

2 . 00 

Phy  Sim 

7.00  5.00 

3.00 

0 . 33 

1  .  00 

1.00 

Tng  Tim 

5.00  3.00 

4 . 00 

0 . 50 

1  .  00 

1  .  00 

The  DM  subjective  weight 

,s 

S 1 1 mul i 

Re3pons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0 . 04 

0.05  0.13 

0 . 35 

0 . 22 

0.21 

The  Ideal  Solution 

S 1 1 mul i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0 . 53 

0.55  0.44 

0 . 49 

0 . 54 

0 . 58 

The  Negati ve- Iuea 1  Solution 

S 1 1 mu 1 i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0 . 04 

0  03  0.06 

0 .04 

0 . 05 

0.08 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Stand  Alone  Trnr  0.54 

Embedded  Tng  0.49 

Instr  Based  Instr  0.48 

Comp  Based  Instr  0.26 


E  -  14 


The  Decision  Matrix 

Instr  Based  Instr 
Comp  Based  Instr 
Stand  Alone  Trnr 
Embedded  Tng 

The  Eigenvector  Pairwise  comparisons 


S timul i 

Respons  Feedbac  Task 

Co  Phy  Sim 

Tng  Tim 

Stimuli 

1 . 00 

1 . 00 

3 . 00 

1  .  00 

1  .  00 

9 . 00 

Respons 

1 . 00 

1 . 00 

1 . 00 

1 . 00 

3 . 00 

9 . 00 

F  eedbac 

0 . 33 

1 . 00 

1 . 00 

5 . 00 

1 . 00 

9.00  6 

Task  Co 

1 . 00 

1 . 00 

0 . 20 

1  .  00 

1  .  00 

1  .  00 

Phy  Sim 

1 . 00 

0 . 33 

1 . 00 

1  .  00 

1 . 00 

1 . 00 

Tng  Tim 

o.n 

0.11 

0.11 

1 . 00 

1  .  00 

1  .  00 

The  DM  subjective 

weights 

S  t i mu 1 i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0 . 24 

0 . 23 

0 . 23 

0  .  12 

0.12 

0 . 05 

The  Ideal  Solution 

S  t i mu 1 i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0.58 

0 . 58 

0 . 58 

0 . 58 

0 . 58 

0 . 58 

The  Negative-Ideal 

So  1 ut ion 

S 1 1 mu 1 i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0.06 

0.06 

0.06 

0 . 06 

0 . 06 

0 . 06 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Instr  Based  Instr  1.00 

Comp  Based  Instr  0.38 

Stand  Alone  Trnr  0.08 

Embedded  Tng  0.00 

s^r-  *  3 

Task  c 


S 1 1 mu 1 l 

Respons 

Feedbac 

Task  Co 

Fhy  Sim 

Tng  7i 

0. 58 

0. 58 

0 . 58 

0 . 58 

0  .  58 

0 . 58 

0 . 26 

0 . 26 

0 . 26 

0 . 26 

0  .  26 

0 . 26 

0.10 

0  .  10 

0.10 

0.10 

0.10 

r\  *  r. 

0 . 06 

0 . 06 

0 . 06 

0 . 06 

0 . 06 

0 . 06 
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The  Decision  Matrix 


S  t i mu 1 


Instr 

Based 

Ins  i 

lr 

0 

.  33 

Comp 

Based 

Ins  i 

tr 

0 

.  47 

Stand  Alone 

Tri 

ir 

0 

.  14 

Embedde 

d  Tng 

0 

.  06 

The 

Ei*. 

anvec' 

tor  Pa 

l  rwi  s  e 

compar  3 

l  sons 

Si 

t i mu 1 i 

Respons  Feedbac  Task 

Stimuli 

1 

.00 

1 . 00 

0 

.  33 

0 

.  33 

Respons 

1 

.  00 

1 . 00 

1 

.  00 

0 

.  33 

Feedbac 

3 

.00 

1 .00 

1 

.  00 

0 

.  33 

Task 

Co 

3 

,  00 

3.00 

3 

.00 

1 

.  00 

Phy 

Sim 

1 

.  00 

1 . 00 

1 

,  00 

1 . 

.  00 

Tng 

Tim 

1 

.00 

3 . 00 

1 

.  00 

3  . 

,  00 

The 

DM  j 

sub j ect i ve 

weights 

St imul i 

Respons  Fe 

edbac  Task 

Co 

Phy  £ 

aim 

0 

.11 

0 

.  1 1 

0. 16 

0 

.  24 

0 

.  15 

The  Ideal  Solution 

Stimuli  Reapons  Feedbac  Task  Co  Phy  Sim 
0.47  0.40  0.44  0.64  0.45 

The  Negative- Ideal  Solution 
Stimuli  Reapons  Feedbac  Task  Co  Phy  Sim 
0.06  0.08  0.07  0.06  0.05 


Respons 

Feedbac 

Task  Co 

Phy  Six 

Tr.g  T. 

0.31 

0 . 42 

0 . 64 

0 . 05 

0 . 07 

0 . 40 

0 . 44 

0 . 23 

0 . 05 

0 . 06 

0 . 08 

0 . 07 

0 . 06 

0 . 45 

0 . 3S 

0 . 22 

0 . 07 

0 . 08 

0.45 

0 . 46 

Co  Phy  Sim 

Tng  Tim 

1  .  00 

1 . 00 

1 .00 

0 . 33 

1 . 00 

100  c-./a 

1 . 00 

0.33 

1 .00 

1 . 00 

1 . 00 

1 . 00 

Tng  Tim 
0 . 24 


Tng  Tim 
0 . 46 


Tng  Tim 
0.07 


Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Instr  Based  Instr  0.56 

Embedded  Tng  0.44 

Comp  Based  Instr  0.40 

Stand  Alone  Trnr  0.39 


4  H 

Task.  c. 
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The  Decision  Matrix 

Instr  Based  Xnstr 
Comp  Based  Instr 
Stand  Alone  Trnr 
Embedded  Tng 


The  Eigenvector  Pairwise  comparisons 

Stimuli  Respons  Feedbac  Task 

Co  Phy  Sim 

Tng 

Stimuli  1.00  1.00 

5 . 00 

4 . 00 

3 . 00 

5 . 00 

Respons  1.00  1.00 

0 . 25 

0 . 25 

0 . 50 

0 . 25 

Feedbac  0.20  4.00 

1 . 00 

5 . 00 

5 . 00 

5 . 00 

Task  Co  0.25  4.00 

0 . 20 

1 . 00 

1 . 00 

3 . 00 

Phy  Sim  0.33  2.00 

0 . 20 

1 .00 

1  .oc 

5 . 00 

Tng  Tim  0.20  4.00 

0 . 20 

0 . 33 

0 . 20 

1 . 00 

The  DM  subjective  weights 

Stimuli  Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0.34  0.09  0.27 

0. 12 

0  .  12 

0 . 07 

The  Ideal  Solution 

Stimuli  Respons  Feedbac 

Task  Co 

Phy  Sira 

Tng  Tim 

0.59  0.55  0.63 

0 . 50 

0 . 67 

0 . 67 

The  Negat ive- Ideal  Solution 

Stimuli  Respons  Feedbac 

Task  Co 

Phy  Sira 

Tng  Tim 

0.05  0.06  0.06 

0.  13 

0.11 

0.11 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Instr  Based  Instr  0.31 

Comp  Based  Instr  0.10 

Stand  Alone  Trnr  0.03 

5/^  *  £  -  - 

T/^SK  c 


Stimuli  Respons  Feedbac  Task  Co  Thy  Sim  T r.g  Tir. 


0 . 26 

0 . 25 

0  .  19 

0 . 23 

0  11 

0 

0.10 

0 . 06 

0.13 

0  .  14 

v  .  1  1 

0 

0 . 05 

0.14 

0 . 06 

0.13 

0  .  l  i 

0 

0 . 59 

0 . 55 

0.63 

0 . 50 

0 . 6  / 

J 
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The  Decision  Ma tnx 


Stimuli 

Respons 

Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

Instr 

Based  Instr 

0 .05 

0 . 05 

0.12 

0 . 20 

0 . 05 

0 . 25 

Comp 

Eased  Instr 

0.15 

0  .  17 

0.13 

0 . 20 

0.19 

0.13 

Stand  Alone  Trnr 

0 . 35 

0 . 28 

0.31 

0 . 40 

0 . 38 

0 . 06 

Embedded  Tng 

0 . 46 

0.51 

0.44 

0 . 20 

0 . 33 

0 . 56 

The  Eigenvector  Pairwise  comparisons 

St imul  i 

Respons  Feedbac  Task 

Co  Phy  Sim  Tng  Tim 

S 1 1 mul i 

1 . 00 

1.00  0.20 

0 . 33 

0 . 33 

0.33 

Respons 

1 . 00 

1.00  0.20 

0 . 33 

1 . 00 

0 . 33 

Feedbac 

5.00 

5.00  1.00 

3 . 00 

3 . 00 

3 . 00 

Task  Co 

3.00 

3.00  0.33 

1 . 00 

1  .  00 

1 . 00 

Phy  Sim 

3  00 

1.00  0.33 

1 . 00 

1 .00 

1  .  00 

Tng  Tim 

3.00 

3.00  0.33 

1 . 00 

1 . 00 

1 . 00 

The  DM  subjective  weights 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.06  0.08  0.40  0.16  0.14  0.16 


The  Ideal  Solution 

Stimuli  Kespons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.46  0.51  0.44  0.40  0.38  0.56 

The  Negati ve- Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.05  0.05  0.12  0.20  0.05  0.06 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  0.82 

Stand  Alone  Trnr  0.52 

Instr  Based  Instr  0.15 

Comp  Based  Instr  0.15 
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1 . 00 
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0 . 33 

2 . 00 

4 . 00 

*sk  Co 

7 . 00 

3.00 

3 . 00 

: .  oo 

5 . 00 

5 . 00 

r»  y  Sim 

0  Z  3 

0.33 

0.33 

0 . 20 

1 . 00 

3 . 00 

ng  Tim 

*\  ic 

0.25 
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0 . 39 
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Che  Idea 
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Z t i mu 1 i 
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Phy  Sim 

Tng  Tim 

0 . 25 

6 . 49 

0 . 49 

0 . 40 

0 . 38 
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1 t i mu 1 i 
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Phy  Sim 
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0 . 25 
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0 . 20 
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0.11 

Rank  Order:  Relative  Closeness  to  -..he  Ideal  Solution 
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Instr  Based  Instr 
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0.38 
0  .  18 
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Z 1 1  mu 
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v-spons 

i  .  00 

: . 00  0.33 

0 . 20 
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.'ask  Co 

7 . 00 

5.00  3.00 

.  V 

! 

•  h  y  Sim 

7 . 00 

5.00  3.00 

0 . 33 

. 

n  g  Tim 

5 . 00 

3.00  4 . 00 

0 . 50 

1  •  0  '* 

" he  DM 

subjective  weights 

'  *  l  mu  1 1 

Respons 

Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0 . 04 

0.05 

0.13  0.35 

*  0 . 22 

0.21 

7he  Ideal  Solution 

!  1 1  mu  1  i 

Respons 

Feedbac  Task  Co 

Phy  Sim 

T.g  Tim 

0 . 53 

0 . 55 

0.44  0.47 

0 . 53 

0.56 

’he  Negat i ve - Idea  1  Solution 

‘  t  i  mu  1  i 

Respons 

Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0 . 04 

0 . 03 

0.06  0.04 

0 . 05 

0.08 

•:ank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  0.71 

Stand  Alone  Trnr  0-56 

Instr  Based  Instr  0.37 

Comp  Based  Instr  0.12 
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The  Decisicn  Matrix 


Stimuli  Res  pons  Feedbac  Task  Co  Phy  Sim  Tr  t  Tirr 


Instr 

Based 

I  r.  s  t  r 

0 . 04 

0 . 05 

0 . 05 

0 . 05 

0  05 

0 . 05 

Comp 

Based 

Ins  tr 

0.14 

0.13 

0  .  13 

0  .  13 

0.13 

0.13 

Stand  Alone 

Trnr 

0 . 35 

0 . 34 

0 . 34 

0 . 34 

0.34 

G  .  34 

Embedded  Tng 

0 . 46 

0 . 48 

0 . 48 

0 . 48 

0.46 

0 . 40 

The  Eigenvector  Pairwise 

compar isons 

Stimuli 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

Stimuli 

1 . 00 

1 . 00 

3.00 

1 . 00 

1 . 00 

9 . 00 

Respons 

1 .00 

1 . 00 

1  .  00 

1 . 00 

3 . 00 

9 . 00 

Feedbac 

0 . 33 

1 . 00 

1 . 00 

5.00 

1 . 00 

9  .  „0 

Task  Co 

1 . 00 

1 .00 

0.20 

1  .  00 

1  .  00 

1  .  00 

Phy  Sim 

1 . 00 

0 . 33 

1 . 00 

1 .00 

1  .  00 

1  .  00 

Tng  Tim 

0.11 

0.11 

0.11 

1 . 00 

1 . 00 

1 . 00 

The  DM  subjective  weights 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.24  0.23  0.23  0.12  0.12  0.05 

The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.46  0.40  0.48  0.48  0.48  0.40 

The  Negat ive - Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.04  0.05  0.05  0.05  0.05  0.05 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Stand  Alone  Trnr  0.69 

Comp  Based  Instr  0.20 

Instr  Baaed  Instr  0.00 
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The  Decision  Matrix 


St i mul l 

Respons 

Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

Tnstr  Based 

Instr 

0 . 39 

0.14 

0 . 36 

0 .44 

0 . 47 

0 . 35 

Comp  Based 

Instr 

0 . 48 

0.44 

0 . 44 

0 . 32 

0.41 

0 . 46 

Stand  Alone 

Trnr 

0 . 06 

0 . 22 

0 . 09 

0 . 09 

0 . 05 

0.14 

Embedded  Tng 

0 . 06 

C  .  2  1 

0.12 

0.15 

0 . 06 

0 . 05 

The  Eigenvector  Pairwise 

comparisons 

Stimuli  Respons  Feedbac  Task 

C;  Phy  Sim  Tng  T 

lm 

S 1 1 mu 1 i 

1.00  1.00 

0 . 33 

0 . 33 

1 . 00 

1 . 00 

Respons 

1.00  1.00 

1 . 00 

0.33 

1 .00 

0 . 33 

Feedbac 

3.00  1.00 

1  .  00 

0 . 33 

1  .  00 

1  .  00 

Task  Co 

3.00  3.00 

3 . 00 

1 . 00 

1 . 00 

0 . 33 

Phy  Sim 

l .00  1.00 

1 .00 

1 . 00 

1 .00 

1 . 00 

Tng  Tim 

1.00  3.00 

1 . 00 

3 . 00 

1 . 00 

1 . 00 

The  DM  subjective  weights 

Stimuli 

Respons  Feedbac  T 

ask  Co 

Phy  Sim 

Tng  Tim 

0.11 

0.11  0.10 

0 . 24 

0  .  15 

0 . 24 

The  Ideal  Solution 

St imul i 

Respons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0.40 

0.44  0.44 

0 . 44 

0 . 47 

0. 46 

The  Negative-Ideal  Solution 

Stimuli 

Respons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0 . 00 

0.U  0.09 

0.09 

0.05 

0.05 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Comp  Based  Instr  0.83 

Instr  Based  Instr  0.75 

Stand  Alone  Trnr  0.13 

Embedded  Tng  0.10 

5/,;t  *V  _ 

TASK  D 


E  -  22 


The  Decision  Matrix 


St  l  mui  l 

Respons 

Feedbac 

Task  Co 

Phy  Sim 

T  n  g  Tim 

Instr 

Based 

I  ns  t  r 

0. 25 

0  .  18 

0 . 22 

0.13 

o .  ;  2 

0.16 

Comp 

Based 

Instr 

0.13 

0.13 

0.12 

0  .  13 

0.13 

0.16 

Stand  Alone 

Trnr 

0.06 

0 . 09 

0 . 08 

0  .  13 

0.13 

0 . 07 

Embedded  Tng 

0 . 56 

0.61 

0 . 59 

0 . 63 

0 . 63 

0 . 60 

The  Eigenvector  Pairwise 

compar isons 

Stimuli 

Respons  Feedbac  Task 

Co  Phy  Sim  Tng  Tim 

Stimuli 

1 . 00 

1.00 

5 . 00 

4 . 00 

3 . 00 

5 . 00 

Respons 

1 . 00 

1 . 00 

0 . 25 

0 . 25 

0 . 50 

0.25 

Feedbac 

0 . 20 

4 . 00 

1 . 00 

5 . 00 

5 . 00 

5 . 00 

Task  Co 

0 . 25 

4 . 00 

0 . 20 

1 . 00 

1 . 00 

3 . 00 

Phy  Sim 

0.33 

2.00 

0 . 20 

1 . 00 

1 . 00 

5 . 00 

Tng  Tim 

0 . 20 

4.00 

0 . 20 

0 . 33 

0 . 20 

1 . 00 

The  DM  subjective  weights 

Stimuli  Reapona  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.34  0.09  0.27  0.12  0.12  0.07 


The  Ideal  Solution 

Stimuli  Reapona  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0 . 50  0.61  0.59  0.63  0.63  0.60 

The  Negative- Ideal  Solution 

Stimuli  Reapona  Feedbac  Task  Co  Phy  Sim  Tng  Tim 

0.06  0.09  0.08  0.13  0.13  0.07 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Instr  Based  Instr  0.31 

Comp  Baaed  Inetr  0.11 

Stand  Alone  Trnr  0.00 
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S 1 1 mu  1 1 

Respons 

Feedbac 

Task  Co 

Phy  Si. 

Instr  Baaed 

Instr 

0 . 22 

0.12 

0 . 06 

0 . 05 

0 . 05 

Comp  Based 

I  ns  tr 

0.12 

0.  13 

0  .  16 

0 . 09 

0 . 06 

Stand  Alone 

Trnr 

0 . 08 

0.31 

0 . 49 

0 . 34 

0 . 4  7 

Embedded  Tng 

0 . 59 

0 . 44 

0 . 29 

0 . 53 

0.41 

The  Eigenvector  Pairwise 

compari sons 

S  t imul l  Respon 

s  Feedbac  Task 

Co  Phy  Sim  Tng  Tim 

Stimuli  1.00  1.00 

0 . 20 

0.33 

0 . 33 

0 . 33 

Respons  1.00  1.00 

0 . 20 

0 . 33 

1 . 00 

0 . 33 

Feedbac  5.00  5.00 

1 . 00 

3 . 00 

3.00 

3 . 00 

Task  Co  3.00  3.00 

0 . 33 

1  .  00 

1 . 00 

1 . 00 

Phy  Sim  3.00  1.00 

0 . 33 

1 . 00 

1  .  00 

1 . 00 

Tng  Tim  3.00  3.00 

0 . 33 

1 . 00 

1  .  00 

1 . 00 

The  DM  subjective  weights 

Stimuli  Respons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0.00  0.08  0.40 

0 . 10 

0  .  14 

0 . 16 

The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0.59  0.44  0.49 

0 . 53 

0 . 47 

0.42 

The  Negat ive- Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co 

Phy  Sira 

Tng  Tim 

0.08  0.12  O.Ofl 

0.05 

0 . 05 

0 . 08 

Rank  Order :  Relative  Cloiantsi  to  the  Ideal  Solution 
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Stimuli 

Re  s  pons 

F  ee Jbac 

Task  Co 

Phy  Sim 

T  n  g 

T: 

Instr 

Based  T 

natr 

0 .04 

0.04 

0 . 04 

C  .  04 

C  .  04 

0 

0  4 

Comp 

Based  Instr 

0.14 

0.14 

0.14 
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Trnr 
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0 . 35 

0 . 36 

0 . 3  5 

0  3b 

0  . 

35 

Embedded 

Tng 
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The  Eigenvector  Pairwise  comparisons 

St  1 mul l 

Reapona 

Feedbac  Task 

Co  Phy  S: 

.  m  Tng  T : 

.  m 

S  t i mul i 

1 . 00 

1 . 00 

3.00 

1  .  00 

1 . 00 

9  -  00 

Reapona 

1 . 00 

1 . 00 

1 . 00 

1 . 00 

3 . 00 

9 . 00 

Feedbac 

0 . 33 

1 .00 

1 . 00 

5 . 00 

1  .  00 

9 . 00 

Task  Co 

1 . 00 

1 . 00 

0.20 

1 . 00 

1 . 00 

1 . 00 

Phy  Sim 

1 . 00 

0 . 33 

1 . 00 

1 . 00 

1 . 00 

1 . 00 

Tng  Tim 

0.11 

0.11 

0.11 

1 . 00 

1 .00 

1 . 00 

The  DM  subjective 

weights 

Stimuli 

Respons  Feedbac  Task  Co 

Phy  Sim 

Tng  Tim 

0 . 24 

0 . 23 

0 . 23 

0  .  12 

0  .  12 

0 . 05 

The  Ideal  Solution 

Stimuli  Reapona  Feedbac  Task  Co  Phy  Sira  Tng  Tim 

0.46  0.46  0.46  0.46  0.46  0.46 


The  Negative- Ideal  Solution 

Stimuli  Reiponi  Feedbac  Task  Co  Phy  Sira  Tng  Tim 

0.04  0.04  0.04  0.04  0.04  0.04 


Rank  Order:  Relative  Cloaeneas  to  the  Ideal  Solution 

Embedded  Tng  1.00 

Stand  Alone  Trnr  0.74 

Corap  Baaed  Inatr  0.24 

Inatr  Baaed  Inatr  0.00 
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The  Decision  Matrix 


S  t  i  mu  1 1 

Respons 

Feedbac 

Task  Co 

Phy  Sim 

Tn  if  T  i  : 

nstr  Based  Instr 

0 . 05 

0 . 30 

0.14 

0.12 

j  .  05 

6.45 

Comp  Based  Instr 

0 . 05 

0  4  7 

0.44 

0.70 

0 . 05 

0  .  25 

Stand  Alone  Trnr 

0  45 

0.12 

0 . 22 

0 . 23 

0 . 47 

0  05 

Embedded  Tng 

0 . 45 

0  .  :  2 

0.21 

0  •  3  7 

0.41 

0.24 

The  Eigenvector  Pairwise  comparisons 


S 1 1 mu 1 1 

Respons  Feedbac  Task 

Co  Phy  Sim 

Tng  1 

Stimuli 

1 . 00 

1 .00 

0 . 33 

0 . 33 

1  .  00 

1  .  00 

Respons 

1 .00 

1 . 00 

1 . 00 

0 . 33 

1  .  00 

0 . 33 

Feedbac 

3 . 00 

1  .  oc 

1 . 00 

0 . 33 

1  .  00 

1  .  00 

Task  Co 

3 . 00 

3 . 00 

3 . 00 

1 . 00 

1  .  00 

0  33 

Phy  Sim 

1 .00 

1 . 00 

1 . 00 

l  .  00 

1 . 00 

1  .  00 

Tng  Tim 

1 .00 

3 . 00 

1 . 00 

3 . 00 

1 . 00 

l  .  00 

The  DM  subjective 

weights 

Stimuli 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0.11 

0.11 

0. 16 

0 . 24 

0  .  15 

0 . 24 

The  Ideal  Solution 

Stimuli 

Respons  Feedoac 

Task  Co 

Phy  Sim 

Tng  Tim 

0.45 

0 . 47 

0 . 44 

0 . 37 

0 . 47 

0. 45 

The  Negat i ve- Ideal 

So lufc ion 

S 1 1 mul l 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

0 . 05 

0. 12 

0 . 14 

0.12 

0 . 05 

0.05 

Rank  Order:  Relative  Closeness  to  the  Ideal  Solution 

Embedded  Tng  0.58 

Comp  Based  Inatr  0.51 

Inatr  Baaed  Inatr  0.47 

Stand  Alone  Trnr  040 
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The  Decision  Matrix 


Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 


Instr 

Based 

Instr 

0 . 06 

0 . 05 

0 . 05 

0 . 22 

0 . 05 

0 . 05 

Comp 

Based 

Instr 

0.41 

c .  : : 

0.21 

0.11 

0.14 

0.21 

Stand  Alone 

Trnr 

0.41 

0 . 24 

0.13 

0  5  1 

0.51 

0.61 

Embedded  Tng 

0.13 

0 . 60 

0.61 

0.17 

0.20 

0.13 

The  Eigenvector  Pairwise 

compar isons 

S  t i mu 1 i 

Respons  Feedbac 

Task  Co 

Phy  Sim 

Tng  Tim 

S 1 1 mu 1 i 

1 . 00 

1 . 00 

5 . 00 

4 . 00 

3 . 00 

5 . 00 

Respons 

1 . 00 

1 . 00 

0 . 25 

0 . 25 

0 . 50 

0 . 25 

Feedbac 

0 . 20 

4 . 00 

1 . 00 

5 . 00 

5 . 00 

5 . 00 

Task  Co 

0 . 25 

4 . 00 

0. 20 

1 . 00 

1 . 00 

3 . 00 

Phy  Sim 

0.33 

2 . 00 

0 . 20 

1 . 00 

1  .  00 

5 . 00 

Tng  Tim 

0 . 20 

4 . 00 

0 . 20 

0 . 33 

0 . 20 

1 . 00 

The  DM  subjective  weights 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.34  0.09  0.27  0.12  0.12  0.07 

The  Ideal  Solution 

Stimuli  Respons  Feedbac  Task  Co  Phy  Sim  Tng  Tim 
0.41  0.60  0.61  0.51  0.61  0.61 
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APPENDIX  F:  BORDA  AGGREGATE  RANKINGS 

This  appendix  provides  the  aggregate  rankings  of  the 
six  individual  subject  matter  experts  for  each  of  the 
e i ght  tasks . 
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APPENDIX  G.  CONSISTENCY  INDEX  DATA  ANALYSIS. 

Recapitulation  of  Consistency  Scores . G  -  2 

Summary  Statistics  .  G-3 
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Cumulative  Distribution  (Interval  0  to  .1)  .  .  .  G  -  6 

Cumulative  Distribution  (Interval  0  to  .05)  .  .  .  G  -  7 

Ordered  Array  of  Consistency  Scores  .  G  -  8 
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SUMMARY  STATISTICS 

NUMBER  OF  VALUES  =  294 
ARITHMETIC  MEAN  =  .124939 
STANDARD  DEVIATION  =  .21161 

VARIANCE  =  4 . 47788E-2 


IF  THE  DATA  IS  CONSIDERED  TO  BE  A  SAMPLE  OF  A  POPULATION, 

THE  FOLLOWING  CAN  BE  INFERRED  ABOUT  THE  POPULATION: 

ESTIMATED  STANDARD  DEVIATION  OF  THE  POPULATION  =  .211971 

ESTIMATED  POPULATION  VARIANCE  =  4.49316E-2 
STANDARD  ERROR  OF  THE  MEAN  =  1.23624E-2 


CONFIDENCE  LIMITS  ON  POPULATION  MEAN: 
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2 .03 

DECILE  RANGE 

= 

.  275 

SEMI -QUARTILE  RANGE 

= 

.112 

PEARSON  SKEWNESS 

= 

.  864 
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DO  YOU  WISH  TO  SPECIFY  ANOTHER  CLASS  INTERVAL  ?  yes 
PLEASE  SPECIFY  LOWER  AND  UPPER  LIMITS  FOR  ANY  TYPICAL  CLASS 
INTERVAL,  I.E.,  FOR  A  TEN  UNIT  INTERVAL  FROM  100  TO  110, 
INPUT  '100,110’. 

?  .0,-1 


FREQ 

UENCY  DISTRIBUTION 

FROM 

UP  TO  BUT 

NOT  INCLUDING  FREQUENCY 

PERCENT 

FREQUENCY 

-  .  2 

-  .  1 

0 

0 

-  .  1 

4 . 44089E-  16 

0 

0 

4 . 44089E-  1 6 

.  1 

187 

63.605 

.  1 

.  2 

62 

21 .088 

.  2 

.3 

20 

6.803 

.3 

.4 

10 

3.401 

.  4 

.  5 

1 

.  34 

.  5 

.  6 

5 

1.701 

.6 

.7 

4 

1.361 

.  7 

.8 

1 

.  34 

.  8 

.9 

0 

0 

.  9 

1  . 

0 

0 

1  . 

1  .  1 

1 

.34 

1  .  1 

1 . 2 

1 

.  34 

1 . 2 

1.3 

0 

0 

1 . 3 

1 . 4 

0 

0 

1 . 4 

1.5 

0 

0 

1 . 5 

1.6 

0 

0 

1.6 

1 . 7 

0 

0 

1 . 7 

1 . 8 

0 

0 

1.8 

1 . 9 

1 

.  34 

1 . 9 

2  . 

0 

0 

2. 

1 . 05 

1 

.  34 

1 . 05 

2 . 2 

0 

0 

2 . 2 

1.15 

0 

0 

DO  YOU  WISH  TO  SPECIFY  ANOTHER  CLASS  INTERVAL  ’  no 
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?  .0.  .05 


FREQ 

U  E  N  C  Y  DISTRIBUTION 

UP  TO  BUT 

PERCENT 

FROM 

NOT  INCLUDING  FREQUENCY 

FREQUENCY 

-  .  1 

-.05 

0 

0 

-.05 

2 . 22045E- 16 

0 

0 

2 . 22045E-  16 

.05 

99 

33.673 

.05 

.  1 

88 

29.932 

.  1 

.  15 

40 

13.605 

.  15 

.  2 

22 

7 . 483 

.  2 

.  25 

8 

2.721 

.25 

.3 

12 

4 . 082 

* 

.35 

7 

2 . 381 

.35 

.  4 

3 

1 .02 

.  4 

.45 

1 

.34 

.45 

.5 

0 

0 

.  5 

.55 

3 

1 . 02 

.  55 

.  6 

2 

.68 

.6 

.65 

2 

.68 

.65 

.7 

2 

.  68 

_  •» 

.  75 

0 

0 

.75 

.8 

1 

.  34 

.8 

.85 

0 

0 

.85 

.  9 

0 

0 

.9 

.95 

0 

0 

.95 

1  . 

0 

0 

1  . 

1.05 

1 

.  34 

1 . 05 

1  .  1 

0 

0 

1  .  1 

1  .  15 

1 

.  34 

1  .  15 

1.2 

0 

0 

1 . 2 

1 . 25 

0 

0 

1 . 25 

1.3 

0 

0 

1 . 3 

1 .35 

0 

0 

1 . 35 

1  .  4 

0 

0 

1 . 4 

l  .  45 

0 

0 

1 . 45 

1 . 5 

0 

0 

1 . 5 

1 . 55 

0 

0 

1 . 55 

1 .6 

0 

0 

1 . 6 

1 .65 

0 

0 

1 . 65 

1 . 7 

0 

0 

1 . 7 

1 .75 

0 

0 

1 .75 

1 . 8 

0 

0 

1 . 8 

1  .  85 

1 

.  34 

1  .  85 

1 . 9 

0 

0 

1  .  9 

1 . 95 

0 

0 

1 . 95 

2  . 

0 

0 

2  . 

0 

1 

.  34 

0 

2  .  1 

0 

0 

2  .  1 

0 

0 

0 
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DO  YOU  WISH  TO  SPECIFY  ANOTHER  CLASS  INTERVAL  ?  yes 
PLEASE  SPECIFY  LOWER  AND  UPPER  LIMITS  FOR  ANY  TYPICAL  CLASS 
INTERVAL.  I.E.,  FOR  A  TEN  UNIT  INTERVAL  FROM  100  TO  110, 
INPUT  ’ 100 ,110'. 

?  .  0  .  .  1 


CUMULATIVE  DISTRIBUTION 


VALUE 

NUMBER  LESS 
THAN  VALUE 

PERCENT  LESS 
THAN  VALUE 

VARIATE  SUM  -  P 
LESS  THAN  VALUE 

.  1 

0 

0 

0 

4. 44089E- 

16  0 

0 

0 

.  1 

187 

63 . 605 

19 . 942 

.  2 

249 

84.694 

43.412 

.  3 

269 

91 . 497 

56.904 

.  4 

279 

94 . 898 

66.258 

.  5 

280 

95.238 

67.347 

.  6 

285 

96.939 

74.515 

.  7 

289 

98.299 

81 . 498 

.8 

290 

98.639 

83.641 

.  9 

290 

98.639 

83.641 

1  . 

290 

98.639 

83.641 

1  .  1 

291 

98 . 98 

86 . 445 

1  .  2 

292 

99.32 

89.519 

1 . 3 

292 

99 . 32 

89.519 

1 . 4 

292 

99 . 32 

89.519 

1 . 5 

292 

99 . 32 

89.519 

1 . 6 

292 

99 . 32 

89.519 

1 . 7 

292 

99 . 32 

89.519 

1 .8 

292 

99 . 32 

89.519 

1 . 9 

293 

99 . 66 

94 . 473 

2  . 

293 

99 . 66 

94 . 473 

1 . 05 

294 

100 

1  00 

2 . 2 

0 

98 . 98 

0 

DO  V0U  WISH  TO  SPECIFY  ANOTHER  CLASS  INTERVAL  no 
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->  .  0  ,  .  05 


CUMULATIVE  DISTRIBUTION 


VALUE 

NUMBER  LESS 
THAN  VALUE 

PERCENT  LESS 
THAN  VALUE 

VARIATE  SUM  -  P' 
LESS  THAN  VALUE 

.  05 

0 

0 

0 

2 . 22045E- 

16  0 

0 

0 

.  05 

99 

33.673 

3.008 

.  I 

187 

63 . 605 

19.942 

.  15 

227 

77.211 

32 . 979 

.  2 

249 

84.694 

43.412 

.  25 

257 

87.415 

48  .  149 

.  3 

269 

91 . 497 

56 . 904 

.35 

276 

93.878 

63 . 076 

.  4 

279 

94.898 

66 . 258 

.  45 

280 

95 . 238 

67.347 

.  5 

280 

95 . 238 

67.347 

.  55 

283 

96.259 

71 . 504 

.  6 

285 

96.939 

74.515 

.  65 

287 

97.619 

77.937 

.  7 

289 

98.299 

81 . 498 

.75 

289 

98.299 

81 . 498 

.  8 

290 

98.639 

83.641 

.  85 

290 

98.639 

83.641 

.9 

290 

98.639 

83.641 

.95 

290 

98 . 639 

83.641 

1  . 

290 

98.639 

83.641 

1 . 05 

291 

98.98 

86 . 445 

1  .  1 

291 

98 . 98 

86.445 

1  .  15 

292 

99 . 32 

89.519 

1 . 2 

292 

99 . 32 

89.519 

1 . 25 

292 

99 . 32 

89.519 

1 . 3 

292 

99 . 32 

89.519 

1 . 35 

292 

99 . 32 

89.519 

1 . 4 

292 

99.32 

89.519 

1 . 45 

292 

99 . 32 

89.519 

1 . 5 

292 

99 . 32 

89.519 

1  .  55 

292 

99 . 32 

89.519 

1 . 6 

292 

99 . 32 

89.519 

1 . 65 

292 

99 . 32 

89.519 

1 . 7 

292 

99 . 32 

89.519 

1 . 75 

292 

99 . 32 

89.519 

1 . 8 

292 

99 . 32 

89.519 

1  .  85 

293 

99 . 66 

94 . 473 

1 . 9 

293 

99.66 

94 . 473 

1 . 95 

293 

99 . 66 

94 . 473 

2  . 

293 

99.66 

94 . 473 

0 

294 

100 

100 

2  .  1 

0 

0 

0 
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APPENDIX  H.  QUESTIONNAIRE  COMMENTS. 

Comments  from  the  subject  matter  experts  were 
solicited  on  the  last  page  of  the  questionnaire. 

Following  the  listing  of  the  comments  (provided  below)  is 
a  short  discussion  of  the  impact  of  those  comments. 

-  Questionnaire  is  slighted  toward  embedded  training 
because  it  is  described  as  the  panacea  for  all  training 
problems  and  most  significantly,  embedded  training  puts 
troops  on  the  actual  equipment. 

-  The  questionnaire  is  subjective.  A  Collective 
Front  End  Analysis  would  allow  one  to  weigh  and  evaluate 
the  capabilities  of  the  different  training  alternatives 
based  on  the  training  alternative’s  ability  to  train  the 
specific  task. 

-  Need  to  include  the  training  alternative  -  actual 
equipment . 

-  The  tasks  selected  in  this  questionnaire  tend  to  be 
best  taught  on  the  tank.  Given  embedded  training  is  on 
tank  training,  by  default  these  tasks  are  best  taught  on 
the  tank.  Also,  most  tasks  do  not  require  OPTEMPO 
(operating  tempo  -  fuel)  or  ammunition  to  train; 
therefore,  whether  embedded  or  not,  the  task  is  best 
trained  on  the  tank. 

-  The  definitions  for  the  criteria  were  not  clear. 

The  questionnaire  could  have  been  reduced  with  no  loss  of 


H 


1 


information;  the  total  time  to  complete  the  project  was  1 
3/4  hours. 

-  Too  long.  Suggest  survey  of  2  or  3  tasks  and  have 
several  survey  editions. 

-  Limit  the  number  of  criteria  per  survey  -  it  places 
a  greater  burden  on  you  but  responses  have  more 
credibility  and  validity. 

-  Rating  scale  on  each  page  is  needed. 

-  Problem:  I  worked  on  the  package  at  different  times 
during  the  day,  over  a  period  of  3  -  4  days.  The  problem 
I  have  is  being  consistent  with  ratings  from  the  previous 
rating  periods.  That  is  why  I  recommend  shorter  survey 
packets . 

-  Somewhere,  you  need  to  specify  initial  versus 
sustainment  training  .  It  is  very  difficult  to  make 
decisions  about  any  of  these  criteria. 

-  Criteria  are  not  mutually  exclusive  and  choosing 
between  them  in  many  cai -?s  is  not  meaningful.  For 
example,  time  and  its  quality  will  determine  most 
everything  whether  it  is  CAI ,  Full  Function,  ET ,  or 
classroom. 

-  The  criteria  appear  to  be  split  in  2  camps.  The 
first  comes  from  the  basic  5  (e.g.  presentation,  practice, 
feedback,  evaluate,  record).  The  2nd  group  come  also  from 
the  basic  training  milieu  but  they  fall  into  a  different 
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class  of  criteria.  I  would  suggest  a  split  between  the 


two  . 

-  If  this  format  is  to  be  used  in  the  future  you 
could  spare  the  reader  by  Just  posing  the  generic  question 
for  each  group  and  present  a  listing  of  forced  choices. 
Question  _ 


Classroom 

CAI 

Classroom 

Full 

Classroom 

ET 

-  Types  of  tasks  (  collective,  crew,  individual) 
would  also  be  useful  in  determining  the  type  of  learning. 

-  Instructions  ambiguous  at  first.  Recommend  more 
than  one  example  then  review  the  criterion  and  comparison 
table  prior  to  beginning  the  survey. 

The  comments  concerning  initial  versus  sustainment 
training,  the  length  of  the  survey,  and  consistency  of 
responses  over  several  days  are  discussed  in  the 
recommendations  section.  Four  comments  were  directed 
toward  the  criteria.  It  is  important  that  the  criteria  be 
understood  so  that  the  pairwise  comparisons  reflect 
judgments  that  do  not  overlap  two  or  more  criteria.  The 
distinction  between  the  criteria  can  be  emphasized  during 
briefings  held  with  the  subject  matter  experts  (see 
Chapter  IX  for  a  discussion  of  these  briefings).  While 
the  criteria  may  fall  into  two  classes  the  distinction  in 
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this  method  is  not  important  since  what  we  are  trying  to 
emphasize  is  the  criterion’s  ability  to  describe  training 
effectiveness.  The  comment  about  limiting  the  number  of 
the  criteria  is  probably  an  indication  that  the 
questionnaire  should  be  shortened.  Saaty  has  established 
that  the  number  of  criteria  should  not  exceed  nine.  SMEs 
will  understand  the  type  of  task  they  are  evaluating.  A 
specific  indication  will  not  effect  the  application  of  the 
method.  If  the  questionnaire  is  slanted  toward  embedded 
training  it  is  caused  by  the  positive  Impression  held  by 
the  subject  matter  expert  not  because  the  design  of  the 
questionnaire  or  wording  of  the  questions  are  faulted. 

The  actual  equipment  is  not  a  training  alternative  since 
the  equipment  by  itself  does  not  train.  It  is  a  medium 
used  by  an  instructor  or  embedded  training.  The  tasks 
used  in  the  demonstration  were  selected  without  regard  to 
the  training  medium.  However,  since  embedded  training 
uses  the  tank  and  an  embedded  training  screening  process 
was  used  to  identify  the  tasks  it  is  not  inconsistent  or 
detrimental  to  the  method  that  the  tasks  might  be  suited 
to  training  on  the  tank. 
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APPENDIX  I.  PROJECT  MANAGEMENT 


ORGANIZATION: 

The  project  was  organized  as  shown  below: 


!U.S.  Army  Armor!  !ARI  Field!  ! Kansas  ! 
! _ School _ <  i  Uni t_Knox  !  ! University ! 


'Directorate  of  I 

1  Ir*AniDi_sD4_S2ai£iDf 1 


!  Pro J  ect ! _ [Training! _ ! ORSA  ! _ ! New  Systems! 

!  I  I  I  Division  I  I Division !  {Training  ! 

I  IDivision  I 


I _ !  Training  i _ i  Committee! 

I  Technology  I  ! Members _ » 

!  Branch _ ! 

I  performed  the  role  of  the  project  manager.  Project 
coordination  occurred  with  three  major  agencies:  the  U.S. 
Army  Armor  School,  the  U.S.  Army  Research  Institute  (ARI) 
Field  Unit  at  Fort  KnoX ,  and  Kansas  University.  Actual 
data  and  technical  input  to  the  project  came  from  five 
Sub-agencies.  First  the  task  analysis  used  to  produce  the 
task  list  for  the  Future  Armored  Combat  System  was 
obtained  from  the  Training  Division.  Training  device 
strategy  and  training  effectiveness  analysis  expertise  was 
provided  by  the  Operations  Research  and  Systems  Analysis 
Division.  Training  device  technical  expertise  was 
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provided  by  the  Systems  Training  Devices  Branch  of  the  New 
Systems  Training  Division.  Additional  information  on 
training  technology  was  provided  by  the  Training 
Technology  Branch  of  the  ARI  Field  Unit  at  Fort  Knox, 
Kentucky.  Training,  operations  research,  and  overall 
project  guidance  was  provided  by  the  project  committee 
members . 

The  original  project  milestones  are  provided  below: 
Research  -  Jan  1987  -  Dec  1988 

Method  Development  -  January  -  February  1989 

Proposal  Presentation  -  29  March  1989 

Collect  Data  -  3-14  April  1989 

Project  Defense  -  3-10  May  1989 

Data  collection  actually  extended  into  early  May  and  the 
final  defense  was  in  late  May.  This  delay  was  caused  by 
the  later  than  expected  return  of  some  questionnaires  and 

i 

by  the  lenghty  data  reduction  required  to  obtain  rankings. 
LESSONS  LEARNED: 

Several  lessons  were  learned  from  this  effort. 

First,  the  matrix  style  organization  was  effective  in  that 
all  agencies  cooperated  and  eventually  produced  both  data 
input  and  comments.  Collection  of  data  would  have  been 
more  effective  had  a  functional  organization  been  possible 
since  direct  authority  could  have  been  exerted  over  the 
subject  matter  experts.  I  was  fortunate  in  that  my  past 
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association  with  the  subject  matter  experts  at  the  Armor 


School  provided  a  spirit  of  cooperation  that  an  outsider 
might  not  have  been  able  to  develop  over  the  short  data 
collection  period.  It  is  likely  that  a  matrix 
organization  would  be  used  in  the  actual  service  school 
setting  since  the  study  analysts  would  have  to  interface 
with  several  school  directorates  to  obtain  the  periodic 
expertise  and  time  from  their  subject  matter  experts. 

Another  lesson  learned  was  that  the  data  computation 
(manipulation)  phase  took  three  times  longer  than 
anticipated.  This  put  tremendous  pressure  on  the 
subsequent  preparation  of  the  final  report  and  defense 
presentation.  Data  collection  should  have  been  completed 
by  March  to  allow  a  month  and  a  half  for  data  manipulation 
and  the  preparation  of  tho  final  report.  The  data  problem 
will  be  even  more  significant  to  a  service  school  agency 
using  the  training  effectiveness  comparison  method  since 
so  many  tasks  are  involved. 

Training  studies  are  frequently  accomplished  by  one 
analyst  over  an  extended  period  of  time.  A  study  of  a 
major  new  system  should  be  conducted  by  a  study  team. 

This  would  facilitate  overall  management  direction  and 
timeliness  of  the  effort.  It  would  assist  with  the  data 
manipulation  problem  since  data  reduction  could  be 
accomplished  by  one  analyst  while  another  continues  the 
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collection  process. 


It  also  provides  workload  scheduling 


flexibility  as  the  lead  analyst  may  be  required  to  perform 
other  tasks  during  several  periods  of  a  several  month 
preliminary  training  effectiveness  analysis. 

Finally  some  slack  time  must  be  provided  in  the  study 
project  schedule.  Key  times  are  the  data  collection  and 
data  reduction  phases  of  the  project.  This  slack  time 
will  mitigate  delaying  effects  of  three  problem  areas. 
First,  subject  matter  experts  often  become  nonavailable 
due  to  unexpected  commitments  to  their  functional 
department.  Second,  subject  matter  experts  whose  data 
produces  an  exceptionally  high  consistency  index  will  have 
to  recompute  their  pairwise  comparisons.  Third,  changing 
workload  priorities  within  the  analyst  team’s  office  often 
reduce  the  time  available  for  data  reduction. 
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